Phase 9 — Crawl and Index Health, and Enhanced Testing

Markers: ▶ being worked on, ⚑ implemented and waiting to be reviewed, ✓ done, ✓✗ closed by deciding not to do it, ? open decision, unmarked = planned.

1. Crawling using synthetic data. A crawl of any size can be run on one machine without fetching anything from the web. The pages are made up as the crawl asks for them, so a run repeats exactly and costs nobody bandwidth.
   1. ✓ A synthetic crawl tab is added to the crawl options. Manage Crawl, then Edit Crawl Options, gains a tab beside the web crawl one. It keeps the settings that still mean something for a made-up web and replaces the ones that do not.
      1. ✓ The settings about reaching the live web are dropped. Repeat type, crawl sleep, robots.txt, restrict sites by url, disallowed sites and seed sites all describe fetching from real hosts. A made-up web has no hosts to be polite to and no addresses to restrict.
      2. ✓ The shape and size of the made-up web are chosen. A graph type says how the pages link to one another. A graph size says how many pages the web holds. A third setting says which pages the crawl starts from, taking the place of seed sites.
      3. ✓ What a page is and what it says are chosen by chance. One model gives the chance a page is html, pdf or an image. Another gives the words that appear on it. The same seed gives the same web every run, so a result can be compared with the one before it.
      4. ✓ Each control on the synthetic crawl tab has a help link and a help page. The web crawl and archive tabs have a help button beside each control that opens a wiki page for it; the synthetic tab had none. Each of its seven controls now has one. The seven pages are Graph Type, Power Law Exponent, Links Per Page, Domain vs Path Links, Document Length, Term Frequency Exponent and Image Probability. They are written as wiki source for Chris to paste through the wiki, since the help pages file is generated and never patched.
   2. ✓ A synthetic crawl makes its own pages rather than fetching them. Where that tab is the one chosen, the fetcher takes its pages from a function that generates them in place of FetchUrl::getPages. Everything after the fetch runs unchanged, so what is measured is the real crawl. A page is a tuple of numbers: a grid page steps one along in each component, a tree page appends each child, a power law page draws its targets by hash. By domain the tuple is one hyphened label, http://i-j.web/, so each page is its own company level domain; by path it sits under http://synth.web/i/j/. The queue server and fetcher see a plain web crawl; only the fetch call differs. Two rules of Yioop's had to give: a host with no dot has no company level domain and is dropped, and five consonants in a row read as spam, so y is now a vowel there. Run as daemons with errors shown: a grid by path reached 66 pages on one robots.txt, by domain 25 pages on one robots.txt per page, and a search of the finished index found a made-up word.
2. Each crawl type is run and what it turns up is fixed. A grid, a tree and a power law web are each crawled under each crawl order, host budgeting and breadth first, with the fetcher and queue server logs read as the crawl runs. A crawl that slows, stalls or stops, or an index that comes back wrong for a search, names the stage at fault, and that is what gets fixed.
   1. ✓ The indexer stores a file of pages in the same time however full its partition is. A power law crawl with eight fetchers ran no faster than with four, at 253 thousand pages an hour. The logs showed the indexer taking longer per file as each partition filled. It went from a few seconds to over thirty and fell back at each new partition. It was searching the whole packed index once per page, only to count new keys. It now keeps the partition's keys in a set built once. On one workload the per-file time had climbed to 1.6 seconds; it now holds at 0.02. The crawl then ran at 830 thousand pages an hour.
   2. ✓ The scheduler hands out work as fast as the fetchers can take it. With the indexer no longer the limit, a crawl of eight fetchers settled at 611 thousand pages an hour with every process idle. The scheduler made one fetch batch a pass and waited five seconds a pass. The rate was the batch size divided by five.
      1. ✓ A scheduler pass waits two seconds, an indexer pass still waits five. Both jobs shared one minimum pass time. The scheduler hands out the work and gains from a shorter pass. The indexer reads a directory of index data and gains nothing. A queue server run from a terminal does both jobs in one process and keeps five.
      2. ✓ A batch short of its size is filled from another tier. A batch whose tier ran dry went out with room left in it, and half the batches were short. The rest is now filled from a tier holding urls. Which tier a batch is made from is still chosen once per batch. That choice is what sends the most interesting outstanding sites out first.
      3. ✓ A dictionary lookup whose file a merge moved finds the key anyway. A merge writes the rows of a tier into the tier below and then takes the tier away. A reader that read the file list a moment earlier asked for a file that had gone. It reported the key absent, so a search lost results. The tiers are now walked again, with nothing locked either way. The lists of files a lookup reads are cached in the process. A list found stale is dropped before the walk is made again, since otherwise the second walk failed as the first did.
      4. ✓ Marking the crawl not stalled no longer warns on every upload. Setting a file's time to an explicit value is allowed only to the file's owner, whatever the permissions say. So the web app warned about a file the queue server owns. Six calls that mean now drop the time argument, which needs only write permission.
   3. ✓ A thumbnail is never larger than its picture. With the indexer and the scheduler no longer the limit, the fetchers were. Their logs alternated runs of passes at 1.4 seconds with runs at 5.5, and the slow runs were batches of images. Each image cost 12 milliseconds, all of it in the thumbnail. A 240 by 60 picture was scaled up to 414 by 256 and then encoded. A thumbnail now fits inside its picture. A hundred images went from 1.24 seconds to 0.13 here. The picture also keeps its proportions inside the thumb, centered on its own average color. Before, it was stretched to the thumb's shape. And a fetch batch now spreads its images evenly among its pages. A fetcher's passes then sit near their two second floor. Before, short passes slept and long ones did not.
   4. ✓ A finished synthetic index is checked through the search API. A synthetic page is rebuilt from its number and the crawl's seed. So what the index says of a page can be checked against the page itself. tests/SyntheticIndexExperiment.php samples pages. It checks that each carries its own title, is found by its rarest term, and is stored once. It checks each is listed as a linker of its targets. It reads document counts by term rank and in-link counts. It reports records stored against pages visited. Run against an eleven thousand page power law crawl made here, every check passed.
   5. ✓ Index and storage faults found by synthetic crawls are fixed. Each came out of running the search experiment over a crawl and reading what the index answered. The faults are of one shape: a cache or a list keyed by a name that gets reused.
      1. ✓ The LSM tree reads a tier file again once it has been rewritten. Its cache was keyed by file name alone. A merge writes a tier of the same name, so a term's rows came back as its first partitions repeated. The cache keeps each file's size and time. It checks them at most once a second, since a query makes thousands of reads. A test drives the tree as the index does.
      2. ✓ A dictionary row is no longer lost when a flush meets a file or a buffer that already holds its key. A flush wrote over the rows of a file named for the same first key. Two rows for one key in one buffer were read as one. Both are merged now, and two cases fail on the old code.
      3. ✓ A dictionary record is read to its own end, and a corrupt posting length names its term. The record read took the end position as a length, so every lookup decoded the records after it too.
      4. ✓ Partition postings stay under their own numbers and in memory. The open postings files were renumbered when the list was trimmed, so a read took another partition's postings. A run with no memory limit held no partition indexes at all. The count came from dividing a limit of -1.
   6. ✓ The synthetic index experiment says what it asks and how long each answer took. It is the tool that found the faults above, so its report was made to show where an answer went missing.
      1. ✓ Each question is one query for one result, timed. Two checks asked for two hundred results and looked through them. They now ask for the term and the url together, and the report gives the seconds each kind of question cost.
      2. ✓ A link the index does not answer for is asked about four more ways. The row says whether the link record is missing or only its url form is.
      3. ✓ Its usage and its checks say what they ask. It names both arguments and says how far it has gone every ten pages. The one-record check counts none and several apart. Term ranks past the crawl's vocabulary are left out.
   7. ✓ A day-long synthetic crawl runs without noise or idle time.
      1. ✓ Four warnings from the day-long crawl log are silenced. They came from a page alias with no group and a tier merged away before its read. The others were a missing text resource and the SSL: Success notice.
      2. ✓ A fetcher pass takes at least one second, not two. On the synthetic web the second second was idle time. The three curl_multi trace lines in FetchUrl go out only with timing on.
   8. ✓ An index is rebuilt in parallel, and a query reaches all of it. A rebuild of a 336 partition crawl left queries answering nothing. The work below found why and changed how indexes are built and read.
      1. ✓ A rebuild runs a pool of builders and one merger that adds partitions in order. Leftover done files are cleared first, and a partition left without its four files is built again.
      2. ✓ A running server sees a dictionary another process rebuilt. It kept its copy of each tier file and its highest tier number, so it answered from the dictionary as it was before.
      3. ✓ A word is read in batches, so a query reaches every partition. A query read only a word's first twenty partitions; it now reads the next twenty when it runs out.
      4. ✓ A guessed query reads one combined key per word. An index built now files each word also under its language and safe search values. A guessed query then needs no intersection with the lang: and safe: words. An older index gets a one-second bound on its batch reads.
      5. ✓ Fetchers build closed partitions and the Indexer merges them in order. Fetchers on the queue server's machine build between batches. The Scheduler holds back batches while more than four wait. A stopped crawl finishes its closed partitions.
      6. ✓ A fetcher builds a partition only when it holds no schedule. A synthetic crawl fell from about a million pages an hour to about 300,000. A fetcher had built a partition for three minutes while holding schedule 529. The queue server drains results in schedule order, so no batch was made from 18:07 to 18:09.
      7. ✓ A crawl shows it is stopping as soon as stop is clicked. Until a queue server reads the stop, the crawl status shows Shutting Down Queue, the first stopping step. The queue server's own steps replace it as they are written.
      8. ✓ Fetchers merge partitions into the dictionary. Between batches a fetcher builds one closed partition and then merges the next built one, under a single merge claim. The Indexer merges only when stopping or when fetchers fall behind, and no DictionaryUpdater runs.
         1. ⚑ Three sandbox crawls ran side by side for 300 seconds each, with 400 documents per partition and two fetchers.
         2. ⚑ Indexer merges: 3,262 pages, 61 of 61 partitions in the dictionary, stop in 20 seconds.
         3. ⚑ Revert to DictionaryUpdater: 10,529 pages, but only 34 of 230 partitions in the dictionary, stop in 5 seconds.
         4. ⚑ Fetcher merges: 4,721 pages, 109 of 109 partitions in the dictionary, stop in 26 seconds.
      9. ✓ A stopping crawl shows how many partitions are left. Crawl Status reads Closing Crawl Dictionary with a line counting the closed partitions still to add. The counts and peak memory the crawl reported stay shown while it stops.
         1. ⚑ In a sandbox crawl stopped after 150 seconds, the count fell from 25 to 2 over 20 seconds.
         2. ⚑ Peak memory stayed at 37,589,888 and visited URLs at 2,711 throughout the stop.
         3. ⚑ The new line is translated into all 24 other locales.
      10. ✓ Pictures in news results keep their shape. A news thumbnail fills its wide box without being stretched. The box shows the middle of the picture instead.
          1. ⚑ The news result image rule gains object-fit: cover. The square thumbnail is cropped to the box, which drops its gray bands.
          2. ⚑ A web result with a picture wrote its alternate text from an undefined constant. It now uses the result title.
      11. ✓ A stop with a backlog can close with partitions unmerged. The shutdown loop stops waiting after BUILD_CLAIM_TIMEOUT, 600 seconds. Builds take about five minutes, so a backlog of several can be cut off.
          1. ⚑ The wait now counts from the last partition finished. It gives up only after twice BUILD_CLAIM_TIMEOUT with none finished.
          2. ⚑ A scripted backlog finishing one partition every two seconds ran 12 seconds past a 3-second limit. A stuck one stopped at 3 seconds.
      12. ✓ A synthetic tree crawl can seed an empty host. SyntheticWeb::seedUrl gives http://.web/index.html for a tree graph when the domain link probability is 0.5 or more.
          1. ⚑ The empty tuple is now the path host's root page. A new test case fails on the old code and passes now.
   9. ✓ A guessed query reads one combined key per word. Most queries are split into one disjunct per word, each intersected with the lang: and safe: words guessSemantics adds. Those two words are on nearly every page, so the intersection cost the most time.
      1. ✓ An index built now files each word under combined keys too. A word gets a key for its page's short language code with safe:all, and one with safe:true on a safe page. The key is the word's first eight key characters, an underscore and a hash. The combined postings point at the word's own positions.
      2. ✓ A query on such an index reads the combined key. A disjunct with one short lang: word and safe:true or safe:all looks each word up by its combined key. The two meta words are then dropped, so a one-word disjunct needs no intersection.
      3. ✓ An older index keeps working, with a one-second bound. A COMBINED_TERMS flag in archive_info marks an index built this way, and a rebuild from partition 0 sets it. Without the flag, a word stops reading new batches after one second.
      4. ✓ Rare word queries ran about three times as fast. On the 75 partition test crawl they took 0.027 and 0.031 seconds, against 0.075 and 0.103 by the old path. The rebuild took 181 seconds against 172.
      5. ✓ A one-word query reports the word's whole count. A word's rows are read twenty partitions at a time, and the count came from the first batch alone. getWordInfo now adds the rows past the batch to the totals when asked for the remaining total. It sums at most EXACT_WORD_COUNT_PARTITIONS rows, 200, and past that scales by the completed partitions over the partitions read.
      6. ✓ The experiment reports held pages by walk length. Its floor of one url in the out-degree held assumed a crawl in breadth-first order and was not checked. Step 4 now prints how many sampled urls are held at each number of links from the seed.
      7. ✓ The experiment's checks and summary read in plain English. Checks 2 and 4 say what is searched for and what should come back. Each summary line names its check by number and says what passing means, and a skipped check says why.
      8. ✓ Checks 2 and 4 ask queries that stay small as a crawl grows. Check 2 searches a page's five least common words together; check 4 searches link: a target with info: the page. Each prints the query it asked for the first held page.
      9. ✓ The experiment samples pages with a site:all query. Walking the synthetic graph found only 16 of 200 pages held, since the crawl follows its own page ordering. The sample is now the first pages site:all returns. Every check then runs over the pages asked for, or all the index holds if fewer. A check 2 failure now names which of the five words the index does not hold for that page.
      10. ✓ A random walk adds ten deeper pages to the sample. site:all gives only the first pages crawled. The walk from before now looks up each page it stops on with info:. It walks again until ten held pages are found, and every check runs over them too.
      11. ✓ A join of several words finds pages past the first partition. A word's walk kept its last document index on moving to a new partition. An intersection then skipped that partition's pages below the stale index. advanceGeneration now clears it, and a WordIterator case fails without the change.
      12. ✓ A queue server listed by the name server's address is this machine. isSingleLocalhost took a machine listed by address for another computer, so queries went over HTTP to the site itself. UrlParser::isSameMachine now compares hosts and resolved addresses.
      13. ✓ A word's walk reaches partitions past its first batch. getWordInfo gave the active partition's row with every batch, so a walk went from partition 19 to the active one and skipped the rest. The row now comes only with the last batch. Postings file handles are now kept by file, not partition number.
      14. ✓ Check 2 asks for all five words with no:rewrite. Without it, rewriteAsDisjunctiveQuery turns the five words into a search for any of them. Checks 4 and 7 now look for the page among the answers, since a join's reported count is never zero.
      15. ✓ Check 2 leaves out words the parser joins into one term. hyphenateEntities joins two query words when the n-gram Bloom filter answers yes, and it does for some number pairs. The joined term is on no page. Check 2 now skips such a word.
      16. ✓ A partition holds at most MAX_ITEMS_PER_FILE documents. PartitionDocumentBundle started a new partition only past the limit, so each full partition held one extra. The WordIterator clamp then dropped that one's posting. The limit now has one name, MAX_ITEMS_PER_FILE, and the iterators fall back to NUM_DOCS_PER_PARTITION rather than 16384.
   10. ✓ A Bloom filter lookup hashes only as far as it needs. Most n-word-gram lookups in hyphenateEntities fail on an early bit, so contains now makes its md5 digests one at a time.
       1. ⚑ Output is identical on 200 pages in five languages, which took 0.44 seconds instead of 0.56. A test checks the answers agree.
   11. ✓ Profile phrase extraction before making builds faster. On the 97 English help pages, canonicalizing took 48 percent of the time and stemming about 43 percent of the ticks.
       1. ⚑ A per-term stem cache cut the time by 11 percent. It changed term positions in one Spanish page, so it is not in.
       2. ⚑ The Spanish stemmer kept the last word's R2 region, and gave no stem for qué after words such as una. It and the French stemmer now reset it.
       3. ⚑ With that fixed, the stem cache gives identical output on 200 pages in five languages, 19 percent faster.
   12. ✓ Only a share of fetchers build partitions at once. BUILD_FETCHER_FRACTION, 0.25, times the fetchers running on a channel gives the build slots. A fetcher takes a slot file before it builds.
       1. ⚑ With both slots held, a fetcher builds nothing. A slot left past BUILD_CLAIM_TIMEOUT is taken back and released.
   13. ✓ A pdf thumbnail draws a magazine cover as a viewer does. Two covers came out scrambled or blank, and six faults in the page reader were found against a pdftoppm render of the same page and fixed. A large four ink jpeg past four million points is read one shade per block rather than handed to the drawing library, which dropped its black ink. A picture paired with a soft mask is laid through the mask rather than opaque. A composite font is read through its descendant and its letters reached two bytes at a time. Trimming a run keeps the zero byte that begins a low numbered letter, and a color set inside a saved state is restored with it. The character spacing operator sets how far apart letters are laid. A picture named inside a form the page draws counts as one of the page's pictures.
   14. ✓ A rebuild prints how it ended. A rebuild that stopped left no word on the terminal. A shutdown line now names a fatal error, the pool's reason and the last progress. A closed terminal no longer stops it where PHP has pcntl. Its first report named a builder silent for 152 seconds on one document. The indexer now gives up on a document after DOCUMENT_TIME_LIMIT, 10 seconds, and moves on. A document a builder stopped on is skipped by the next builder of that partition.
   15. ⚑ Pages keep more of their text and are stored under the right language. What a crawl keeps of a page can be checked one page at a time with ArcTool test-page.
       1. ⚑ Page processors keep entities, structured data, PDF streams and the text of subset fonts.
       2. ⚑ A page's declared language wins a tie in the language guess.
       3. ⚑ The word filter's Factory Settings drop bot-check pages. Six more kinds of site have built-in scrapers.
   16. ⚑ The fetcher gets what a browser would and sends it promptly. A crawl's results reach the index within a minute of its start.
       1. ⚑ Crawls carry cookies, drop tracking parameters and start from 2026 seed sites. curl's cookie store reads the null device, so its verbose output holds no cookie file warning.
       2. ⚑ A site's root keeps the page of a same-host redirect or of a working default-locale subsite.
       3. ⚑ A crawl starts only with plugins that exist, their names trimmed. A name ending in a newline stopped the worker starting it.
       4. ⚑ An archive crawl sends its first pages at once, then every minute. Uploads go in slices.
   17. ⚑ A crawl spends its time on new pages. The crawl went from about 60,000 to 90,000 pages an hour.
       1. ⚑ A fetch batch goes to one fetcher. A pass counts the hosts it must look up, not urls, and frees its handles in the next pass.
       2. ⚑ Summaries stop scoring once full, and a page is parsed once for scraper signatures. The Chinese taggers read each weight once.
       3. ⚑ A long PDF guesses its language from its first 8,192 bytes. A scanned PDF is read by OCR only until its text fills a summary.
       4. ⚑ A PDF picture goes to OCR as a gray map or as its own JPEG bytes, not drawn point by point. Scans packed by fax or JBIG2 coding, or one bit to a point, are opened first, so their words are read.
       5. ⚑ A fetch pass logs how many of its transfers finished by each second.
       6. ⚑ Fetchers send how long each kind of document takes to process. A kind 100 times slower than a web page is queued two tiers back. The first 1,000 kinds are kept, saved and read back on a restart.
       7. ⚑ A quick look at squares of a picture skips OCR where none is like print. CMYK pictures in a PDF are read.
       8. ⚑ A fetcher packs the pages and links it sends at zlib level 1, and takes them off its list without moving the rest.
   18. ⚑ PDF thumbnails draw JPEG 2000 photos, bit masks and turned text. A thumbnail is made again by a POST, which the five second limit on a GET does not stop.
   19. ⚑ A long dictionary merge no longer stops the crawl. Only closed partitions still to be built hold crawl batches back. In a crawl of about 1,550 partitions, one fetcher merged tiers 0 to 8 from 19:02 to 20:01. The other fetchers got no batch from about 19:25 to 20:03.
   20. ⚑ Fetchers reach a name server named 0.0.0.0. A request FetchUrl::getPage makes to 0.0.0.0 or :: goes to the loopback address, as Linux and macOS do already. Windows refuses 0.0.0.0, which a secure-start install had stored as its name server.
   21. ⚑ Scanned PDF pages decode in seconds, and slivers of pictures are not read. A JBIG2 page of 33 million pixels took 60 seconds and over a gigabyte to decode. It now takes under 5 seconds and under 100 megabytes. Pictures under 8 pixels on a side are no longer handed to tesseract.
   22. ⚑ A cover drawn without pictures gets its own thumbnail. Such a page was refused, and a photograph from a later page stood in for it. Its title letters, colors and single-letter runs now land where the page puts them.
   23. ⚑ A magazine cover's thumbnail is made in a tenth of a second. It took 8 seconds, longer than a page request may run. Finding the catalog and filling each letter now work on the small part they need.
   24. ⚑ A dictionary tier merge writes a fifth of the bytes it did and runs three times faster. Each 32 KB data file made the writer list its folder and rewrite the folder's whole index. The writer now keeps those lists in memory. A merge writes each index once.
   25. ⚑ A dictionary key is found when its file name sorts out of key order. Tier indexes listed file names sorted as URL-encoded text, and = or a byte written as %XX sorts apart from its key. Indexes are now sorted by key when written and read. A lookup finds a key holding the bytes \xFE or \xFF in its escaped form.
   26. ⚑ Guessing a page's language takes under half the time it did. The Turkish stop word list held an empty word, which matched at every word boundary of every page guessed. Chinese normalizing now replaces characters with one strtr call.
   27. ⚑ A star in a robots.txt rule can match no characters. A rule starting with a star got a slash put before it, and each star had to match at least one character. So a disallowed /search was crawled.
   28. ⚑ A word in a page's host, title or path is filed in a first tier of the index. A query for a common word scored only the first 200 postings, in partition order. So a page naming the word in its title could be missed. Such pages are now filed under a field key, and a query reads that key as a disjunct of its own.
       1. ⚑ IndexDocumentBundle adds the field key beside each combined key, in new bundles and in ArcTool rebuilds.
       2. ⚑ PhraseModel adds the disjunct where the index has FIELD_TERMS. UnionIterator reads its short list first.
   29. ⚑ A url holding a null byte no longer stops a fetcher. curl refuses such a url, and the error ended the fetcher's run. Nine such errors were in yioop.com's error log. getPages now sends the byte as %00, in the url and in the referer.
   30. ⚑ A fetcher spends less time on each page it processes. Page processing took 37% of the fetchers' time, and HtmlProcessor 20%. Each change below gives the same output as before on the saved pages.
       1. ⚑ Scraper signatures are tested on the document object model (DOM) HtmlProcessor already builds. Parsing the page a second time for them took a third of HtmlProcessor's time.
       2. ⚑ A matched scraper reuses that DOM for its text path. It parses the page for extract fields only when it has some.
       3. ⚑ JSON-LD text goes after the body's opening tag. The body keeps its attributes, and a dollar amount such as $250 keeps its digits.
       4. ⚑ English stop words are removed by looking each word up in a set. That is about four times faster than the pattern of 676 words.
       5. ⚑ stemCharGramSegment replaces punctuation with Perl Compatible Regular Expressions (PCRE), not Oniguruma. Phrase extraction for indexing takes 26% less time.
       6. ⚑ tesseract runs with one OpenMP thread. Text recognition took a fifth of the time, with the same text read.
       7. ⚑ A tier merge keeps the last key it wrote, so it does not decode the last record for each row.
   31. ⚑ A fetcher downloads its next batch while it processes the current one. In 35% of yioop.com's download rounds, the fetcher sat 9 seconds or more waiting on its last few pages.
       1. ⚑ The next batch downloads in a Fiber, and getPages hands control back on each pass of its wait. The batch is processed in between.
       2. ⚑ A batch downloading this way is held schedule, as to_crawl is. So a fetcher sends its results and asks for a schedule at the same points as before.
       3. ⚑ The download is finished before an upload, and a crawl change drops it with the rest of the old crawl's urls.
       4. ⚑ A test crawl of slow local sites, with 60 ms per page processed, fetched 35% more pages in four minutes.
   32. ⚑ The crawler can sign its requests under Web Bot Auth. A site behind Cloudflare that has accepted a signed bot lets it past the challenge that stopped www.crunchyroll.com.
       1. ⚑ It is on whenever the name server is reached by https, and needs no setting. The name server's https address is the Signature-Agent.
       2. ⚑ The name server keeps the Ed25519 key pair in SECURITY_DIR, beside its other keys, and makes it the first time.
       3. ⚑ A fetcher asks for the key pair in its crawl time request until it has one. The name server sends it only on a request that came by https.
       4. ⚑ getPages signs each https request with Signature-Agent, Signature-Input and Signature headers.
       5. ⚑ The web server serves the public key, signed, at /.well-known/http-message-signatures-directory.
       6. ⚑ The test checks the code against Cloudflare's example key and example signature.
   33. ⚑ A large sitemap is parsed only as far as the links it gives. In the yioop.com fetcher logs, 1,277 gzipped files took about 1,176 seconds. 1,273 of them were istockphoto sitemaps.
       1. ⚑ XmlProcessor parsed each sitemap once for its root element. SitemapProcessor then parsed it again for its links.
       2. ⚑ XmlProcessor now reads the root element's name with a regular expression. SitemapProcessor::entriesPage cuts the text after the entry that gives the last link allowed, and closes the root element.
       3. ⚑ A cut-off sitemap is cut back to its last whole loc element. A piece of url before the cut no longer becomes a link.
       4. ⚑ A ten megabyte sitemap like istockphoto's now takes 0.03 seconds in CompressedProcessor, where it took 1.0 to 1.2. Seventeen sample documents give the same summaries as before.
   34. ⚑ A gzipped page keeps the crawl's link limit. CompressedProcessor made the processor for the uncompressed page with the summarizer option where the link limit goes.
       1. ⚑ That processor took every link, and left 'dk' as every processor's limit. It now gets the limit, then the summarizer option.
   35. ⚑ PackedTableTools reads and writes seven kinds of row correctly. Each fault has a test case that fails on the code before.
       1. ⚑ findRowFromKeyTableString finds a first row whose key holds a byte save writes as two bytes, as a document id may.
       2. ⚑ It finds rows of a table whose keys vary in length.
       3. ⚑ add with ADD_FILE_PATH no longer calls fsize, which PHP does not have.
       4. ⚑ An empty table loads as no rows, not as one row with an empty key.
       5. ⚑ A negative INT of two or more bytes keeps its sign.
       6. ⚑ An INT size byte equal to the character 0 no longer reads every INT column as 0.
       7. ⚑ findEntryAtIndexTableName gives nothing for a row number past the last, not the whole table.
   36. ⚑ A partition builds in less time. Fetchers built partitions for about a third of their time in the logs of October 13.
       1. ⚑ The file naming the document being built was opened and closed for two writes a document. It is now opened once and written over.
       2. ⚑ addTermPostingLists writes each posting's bytes itself, with PackedTableTools::packIntColumn, and makes a word's canonical key once.
       3. ⚑ NWordGrams::ngramsContains keeps up to NGRAM_LOOKUPS_KEPT answers, a new Config.php value, so a phrase met again is not hashed again.
       4. ⚑ hyphenateEntities lowercases each word once and joins short entities without a slice.
       5. ⚑ A 4,000 page English partition built in 39.8 seconds before and 18.5 after. A 10,000 page partition in many languages took 140 and 95.
       6. ⚑ The postings, document map and last entries files came out the same byte for byte.
   37. ⚑ A position list with a long gap reads back as written. encodePositionList wrote some Rice codes wrong, so 3, 6, 2087 read back as 3, 67108868, 67108869.
       1. ⚑ It now gathers bits in an integer and writes them a byte at a time, which also takes about a fifth less time.
       2. ⚑ Of 390,829 lists from the help page build, 10,789 were written differently, and every changed list now reads back as written.
       3. ⚑ A list with a gap of 0 has no gamma code. It is written as before.
   38. ⚑ Searches and the indexer report where their time goes, and the indexer compresses faster. yioop.com's crawl logs showed the indexer idle 0.04% of each hour. In 8.5 minutes it indexed 20 uploads while fetchers sent at least 39.
       1. ⚑ AnalyticsManager::addTiming counts and times each kind of step, and timingsReport lists them.
       2. ⚑ QueryTool and the query statistics list dictionary, postings, positions, document map and latest copy reads.
       3. ⚑ The indexer logs how many index data files wait, and times uncompressing, compressing, writing and saving.
       4. ⚑ GzipCompressor uses ARCHIVE_COMPRESSION_LEVEL, 6, in place of 9. Indexing a 700 page upload took 24% less time.
       5. ⚑ AnalyticsManager.php's docblocks are brought up to standard.
   39. ⚑ Crawl Status has a Crawl Speed History of the crawl's speed over time. The crawl status gave only the rate over the last hour since the queue server started.
       1. ⚑ A media job, CrawlSpeedJob, runs on the name server each CRAWL_SPEED_SAMPLE_INTERVAL, ten minutes, so it needs no queue server there.
       2. ⚑ It reads the crawl status from the queue servers and logs a line for each running crawl to CrawlSpeed.log in the log folder.
       3. ⚑ CrawlModel::crawlStatus reads the lines into each crawl's SPEED_HISTORY.
       4. ⚑ A [Crawl Speed History] link after Most Recent Fetcher shows the rows newest first, in a table of fixed height that scrolls.
       5. ⚑ An open table stays open when the status refreshes, through crawl_toggle.
       6. ⚑ CrawlstatusView.php's docblocks are brought up to standard.
   40. ⚑ The scheduler looks up a fetch batch's etags in bucket order. In yioop.com's logs of 15 October, fetchers sat idle because no schedule was ready. Writing batch 8490 of 1,301 urls took 17.3 s.
       1. ⚑ Fetcher 7 slept on "No data" 18 times in 22 minutes, 180 s of 1,318 s, close to its logged 15% idle.
       2. ⚑ Each url of a batch was looked up in the etag table alone. A lookup reads its bucket's index unless index_buffer holds it, and the buffer is emptied past INDEX_CACHE_SIZE, 100, buckets.
       3. ⚑ LinearHashTable::getMany orders a batch's keys by bucket, so each bucket index is read once. produceFetchBatch uses it and logs the time the lookups took.
       4. ⚑ For 1,300 urls in a 20,000 url table of 64 record buckets, the lookups took 0.021 s against 0.056 s, finding the same 636.
       5. ⚑ A merge of partition 1873 had 9 s with no log line before it began and 8 s after "Switch Partition time". Fetchers now log how long they look for a partition to build. Merges log the time to open the bundle, record the merge and merge tiers.
   41. ⚑ Builds keep the stems of words, and fetchers log the load average. Building a partition took 75.6 s on yioop.com, 31.4 s of it finding phrases. In a profile here, stemming took 13.5% of a build.
       1. ⚑ Tokenizer::stem keeps up to STEMS_KEPT, 50,000, words' stems, and stemWord stems a word not seen before.
       2. ⚑ Building partition 1 of a test index took 46.6 s against 51.9 s, and extractPhrasesInLists 21.3 s against 27.2 s. The partition's files came out the same.
       3. ⚑ HtmlProcessor took about 30% longer per page after fetchers began compressing pages at level 6. "Process pages time" now carries the load average, to show whether other processes held the cores.
       4. ⚑ A fetcher deflated a page's gzip bytes again inside the message, which cannot shrink them. packSeenSite now sends the page apart, after PAGE_APART_MARK and the rest of the site.
       5. ⚑ For 600 sites of 60,000 byte pages, packing took 1.21 to 1.41 s against 1.24 to 1.59 s. Unpacking took 0.003 s against 0.038 s, and the message was 0.8% smaller. The archive's bytes are the same.
   42. ⚑ Builds split phrases into words faster.
       1. ⚑ Tokenizer::stemPhrase splits a phrase with preg_split on runs of Unicode space, where mb_split took five times as long. Eleven phrases with tabs, no-break and other Unicode spaces split the same both ways.
       2. ⚑ With kept stems as well, a build of partition 1 took 49.1 s against 51.7 s with kept stems alone. Its postings file came out the same.
   43. ⚑ The words of an entity are stemmed once. For "cs dept san jose state university", yioop.com ranked other pages above San Jose State University's own pages.
       1. ⚑ A page with "San Jose State University" files the entity san-jose-state-univers. Its words were found by splitting the stemmed entity and stemming each part again. So "university" was filed as "univ", but a query for "university" looks up "univers".
       2. ⚑ The words of an entity are now taken from the entity before it is stemmed. The local test index has no entity whose last word changes when stemmed twice, so its partition 1 came out the same.
       3. ⚑ A word with diacritic marks on Latin letters, such as josé, is also filed without them, as jose. foldDiacritics uses PHP's Normalizer, or a table of Latin letters where intl is missing.
       4. ⚑ Pages already built keep the old words until they are built again.
       5. ⚑ Comments added in the last patch are rewritten so a developer who has not read the rest of the class can follow them.
   44. ⚑ The scheduler's etag lookups search bucket files as strings. On yioop.com, looking up the etags of 226 to 1,664 urls took 2.7 to 3.6 s. The lookups found 0 to 173 in a table of 4,953,658 rows.
       1. ⚑ At 16,384 rows a bucket, the table has about 300 buckets, and index_buffer keeps 100. getMany turned each bucket index it read into an array, which took 4.1 ms for a full bucket here. Searching the file's string for the key took 0.16 ms.
       2. ⚑ getMany now searches the string. A bucket index already in index_buffer is used as it is, since it may hold rows not yet saved. get and getMany share recordFromRow, which turns a found row into a record.
   45. ⚑ Fetchers encode pages while they wait on downloads, not while they send. In the yioop.com fetcher logs, sends took about a fifth of each fetcher's time. Compressing seen urls took 15.1 s a send.
       1. ⚑ A profile of that step on 300 real pages put 82% of it in packSeenSite's gzip encoding of each page.
       2. ⚑ A download round lasted 7.1 s on average, but 90% of its transfers were done by 5.6 s.
       3. ⚑ FetchUrl::getPages calls a function while it waits. The fetcher's compressWaitingPage encodes one waiting page each call.
       4. ⚑ packSeenSite uses those bytes, and the string sent is the same. The send step on the 300 pages took 0.60 s against 4.85 s.
   46. ⚑ Dictionary merges read each term's occurrence count straight from the last_entries file. Merging the three partitions of a test index took 26.0 s against 34.4 s. The dictionary's data files came out the same.
       1. ⚑ A profile put about a quarter of a merge in loading last_entries as an array, then finding and unpacking each term's row.
       2. ⚑ addPartitionPostingsDictionary now reads NUM_OCCURRENCES from each row in one pass over the file. A row of another shape is unpacked in full.
       3. ⚑ The progress line of the merge loop is formatted only when it is written.
   47. ⚑ Builds split words with preg_split and join entities with one pass less. Building partition 1 of a test index took 43.4 and 41.8 s against 44.6 and 44.7 s. Its postings and positions came out the same.
       1. ⚑ stemTermsK split text with mb_split. preg_split on the same space characters gave the same terms in a quarter of the time.
       2. ⚑ hyphenateEntities lower cases the text once, and goes on to the next word after an entity without an empty step between.
   48. ⚑ A rebuild stops a stuck builder and goes on, and mb_ereg calls become preg or string calls. A rebuild of yioop.com stopped at partition 698 when one builder sat 151 s on one document.
       1. ⚑ ArcTool now stops a stuck builder and hands its partition out again. The next builder skips the document the stopped one was on.
       2. ⚑ A stuck merger still stops the rebuild, as a merge run twice could add rows twice.
       3. ⚑ mb_ereg_replace and mb_split calls became preg_replace, str_replace or explode. They were in PhraseParser, PhraseModel, SearchController, UrlParser, the RSS and SVG processors, the word filter, the Greek tokenizer and TokenTool.
       4. ⚑ On 2,028 texts, including every Unicode space character, each new call gave the same result, four to eleven times faster.
   49. ⚑ The Chinese word splitter joins at most 64 characters into one unknown term, so its time grows in step with the text. A page of yjt.xizang.gov.cn held 8,564 characters of base64 text, and building its words took minutes. StochasticTermSegmenter::segmentSentence joins runs of non-Chinese characters, numbers and punctuation into terms. It tested the growing string at each step from every start, so a run of m characters cost m cubed. The walk from each start now stops after MAX_JOINED_LENGTH characters. A run of 8,000 dashes took 0.24 s, against 33 s for 4,000 before.
   50. ⚑ Merges sync the files they write with one file system sync on Linux. A tier merge and a finished merge step synced each data file on its own, waiting for one journal commit per file. UndoLog::syncPaths runs sync -f once on Linux and syncs file by file elsewhere or when that fails. A local tier merge of 3,430 files went from a median of 5.67 s to 4.84 s with the same files.
   51. ⚑ A page whose download fails in a way that may pass is fetched again later. A page with no answer on its second try in a batch was dropped silently. A server error ended a page for the crawl. Fetchers now send such pages back in RETRY_URLS, and the Scheduler queues each again up to FETCH_TRIES_MAX times. A 403, a 404 and other final codes are not retried.
   52. ⚑ Each https connection sends a different TLS ClientHello, and a refused landing or seed page may be tried again. Every https fetch sent the same ClientHello, so a site could block Yioop by its JA3 or JA4 fingerprint. FetchUrl::varyTlsHello now offers a random choice and order of cipher suites and groups, and sometimes asks for HTTP/1.1. With robots.txt set to Allow Landing Page or Ignore, a 403 on a seed or landing page is queued again like a server error.
   53. ⚑ A site whose name leads to the fetcher's own machine is not indexed with that machine's pages. www.tovinothomas.com had the address 127.0.0.1. A fetcher sent it to its own web server and indexed yioop.com's home page under that name. getPages now refuses such a url before connecting when the queue server gave the address. Otherwise it drops the page once curl shows a loopback or 0.0.0.0 address. The page gets code 421, so a robots.txt with it stops the host. A url for one of localHosts(), or for an address, is still fetched.
   54. ⚑ Each crawl's speed history has its own Crawl Speed Data page. A running crawl's [Crawl Speed History] link opens it. So does the Visited/Extracted Urls count of a crawl in the Crawls table. The page heading names the crawl, and a close button returns to Manage Crawls. The manageCrawls argument speedhistory reads the rows through CrawlModel::crawlSpeedHistory, and CrawlspeedElement draws them. The table that opened inside Crawl Status is gone.
   55. ⚑ The Crawl Speed Data table is taller, and the feed crawl's counts are not a link. The table now scrolls past 36rem rather than 24rem. The media updater makes the feed crawl and logs no speed for it.
   56. ⚑ A fetcher that has held a schedule for two minutes sends what it has, so the other fetchers keep getting schedules. The queue server takes schedules back in the order it sent them. In the logs of 18 October one fetcher held schedule 415 for 683 seconds on pages of www.olx.ua. The send window stayed full nearly that long, and fetchers slept 2,030 seconds in it. Fetcher::ANSWER_SCHEDULE_AFTER now makes the fetcher send once the schedule is that old. Later sends of the same schedule are taken as before.
   57. ⚑ A slow HTML page's log line names the steps it spent its time in. HtmlProcessor::process keeps the seconds of each step in $step_times. The Fetcher adds them to its Slow page line. The olx.ua pages took about four seconds each, and the cause is not yet known.
   58. ⚑ A fetcher that stops on a page goes on with its schedule without that page when it restarts. On 18 October fetchers stopped on two lobbying spreadsheets at 16:08, 16:25, 16:42 and 17:16. Each restart dropped the schedule, so the window waited 17 minutes and the next fetcher stopped on the same page. The Fetcher now keeps the url being processed and the schedule's unsent urls in SCHEDULES_DIR. On restart it skips that url and goes on, unless the schedule has been sent elsewhere. A fatal PHP error is now written to the fetcher's log with the page.
   59. ⚑ A PDF gets at most five seconds of picture reading. In the 18 October logs one batch spent 266 of its 294 PDF seconds in tesseract, on two nissan-global.com PDFs of 112 and 136 seconds. getText read every picture while the words found stayed shorter than a summary. It now starts no further reading once a document has had MOST_READING_SECONDS of it. A made-up PDF of 25 charts each holding one word went from 14.9 to 5.8 seconds, and a slide PDF read the same 17,314 characters as before.
   60. ⚑ The five second limit on reading a PDF's pictures is taken out. It did not change the crawl's pages an hour. PDFs past five seconds took at most 454 of 7,723 fetcher seconds in the 18 October logs. That is under 6 percent, within the speed table's own swings. The limit also lost words of picture-heavy PDFs, so it went.
   61. ⚑ Pages with no text are no longer grouped as duplicates, and hyphenated hosts and titles give the hyphenated term. www.pepsi.com's info: entry led to www.cyon.ch's page, and a pepsi search could not find it. Both pages had no text, so the same content hash, and the partition kept only the shorter url's words. computePageHash now hashes such a page by its url. A coca-cola search gave www.coca-cola.com neither its host nor its first title bonus. Host words now keep coca-cola whole, Unicode's hyphens become the plain hyphen, and ownership signs such as the registered sign leave terms. Pages indexed before the patch keep their old terms.
   62. ⚑ A fetch opens no file, so a restarted fetcher no longer stops, and challenge pages and Drupal menus stay out of the index. A restarted fetcher held every connection the web server had open. Under a limit of 1024, its first batch could not open curl's record files. getPages then handed curl false and the fetcher stopped, 226 times on 18 and 19 October. getPages now reads the address and request from curl, so it opens no file, and each fetcher has its own temporary folder. A page holding Imperva's challenge script and no text is fetched again later, not indexed. The Drupal scraper also reads main tags, through database version 131. Page processors now write temporary files in their fetcher's folder. A redirect's link adds the redirecting host's words, such as coke, to the page it leads to. A home page redirect also links the target's home page, which counts those words as host words.
   63. ⚑ A document whose words are not all UTF-8 gets its thumbnail with no deprecation notice. TextProcessor::drawText read the words with a UTF-8 pattern, which gives null on a stray byte. wordwrap was then handed null, a notice on PHP 8.1 and later. The words are now scrubbed to UTF-8 first.
   64. ⚑ Sentence splitting reads a long run of letters once, and a slow page line names the slow part of a summary. Three of ten fetchers spent about 75% of their time summarizing charter.com newsroom pages, at 26 to 49 seconds a page. The sentence end pattern reread a run of letters from each of its letters, so its time grew as the square of the run. From about 7,000 letters on it gave up, and the block's text was lost. The blank line pattern repeated a group once per newline and gave up the same way. Both now take time in step with the text. ScrapeSummarizer times its parts, and HtmlProcessor adds them to the slow page line with block, sentence and byte counts.
   65. ⚑ A home page with no text of its own is indexed by the words of its url when landing pages are allowed. https://www.pepsi.com/ answers with an Incapsula bot challenge, which the fetcher dropped, so the site never reached the index. In a crawl whose robots.txt mode is Allow Landing Page, such a home page is now indexed by its host words on the Scheduler's last try. The schedule's first line lists those urls as LAST_TRY_URLS, so an earlier real copy is not replaced while tries remain. A home page that answers with no text is described the same way and keeps its title. A host that never answers is not indexed.
   66. ⚑ A redirect's record keeps the urls it went through. The fetcher set the LOCATION field to array_push's count, not the list, so the redirect's urls were lost. PhraseParser now gives those urls location: words only. An info: word would let a redirect stand in for its target's page.
   67. ⚑ A page with a megabyte of inline style keeps its text. The whole corporate.charter.com newsroom page has 1.8 MB, mostly inline style. The lazy pattern removing style elements hit pcre.backtrack_limit and returned null, so no text was found. L\removeTagSpans now removes script, noscript and style elements by string search. HtmlProcessor, TextProcessor, Summarizer, FetchUrl and AddressesPlugin use it. On 100,039 test pages it gave what the patterns gave wherever they did not give up.
   68. ⚑ Tokenizers find their shipped lexicons and character map again. The English and Hindi lexicons and the Chinese character map were read only from the site's locale folder. A site with no such folder got no part of speech tags and no simplified characters, so three unit test cases failed. They are now found with localeResource, which falls back to src/locale. The Hindi lexicon's lines end in \r\n, and the \r is no longer kept in a word's tag. The English stemmer case no longer ends the whole test run at its first wrong stem.
   69. ⚑ A fetcher's processing time line names more of its rest. The next batch's download passes and the link checks are timed apart from the rest. In the logs of 21 October the rest was 719 of about 2,900 seconds of processing.
   70. ⚑ A url goes into a fetch batch once. The queue can hold a url more than once, a landing page above all. The Scheduler put every copy in the batch. In the logs of 23 October about one page in ten a fetcher processed was a copy; fetcher 4 processed http://www.jyb.cn/ 39 times in two and a half minutes. produceFetchBatch now keeps the urls a batch holds and drops a copy.
   71. ⚑ A fetcher's send queues each url once. A fetcher sends one copy of a url for each page it found the url on. The url filter removes only urls queued by earlier sends. So every copy of a url new to the filter went into the queue, and later into a fetch batch. processToCrawlArchive now skips a url it already queued from the same send.
   72. ⚑ Chinese text is segmented in about half the time. Since the tokenizers found their shipped files, normalize maps traditional characters to simplified ones. It used strtr, which builds a table of 3,964 characters on each call. The named entity tagger calls it once a word, so it took 40 percent of the time of reading a Chinese description. A short text is now mapped a character at a time. The segmenter also keeps the words of the runs it segmented lately. A description is segmented to be indexed and again to find its questions and answers. A 2,000 character description takes 61 to 72 milliseconds, from 117 to 152.
   73. ⚑ The centroid weighted summarizer times its parts. The crawl uses it, and its slow page lines said only “summary”. They now name the sentences, the term frequencies, the average, the scoring and the putting together.
   74. ⚑ A fetcher spends less of its time summarizing pages and packing them to send. In the crawl logs of 24 October the summarizer took 369 of 410 seconds on 130 slow pages. Packing a send took 815 seconds over 60 sends, mostly working out unused terms.
       1. ⚑ The summarizer counts a sentence's terms from the list it split the sentence into. It searched the sentence for each term, time that grew with length times terms. That search counted no Chinese word and no term such as #tag.
       2. ⚑ A send works out only a page's question answers. extractPhrasesInLists takes a question answers only parameter. A language that cannot answer questions skips the work, and a link no longer gets the answers of the page packed before it. Packing 2,000 pages of eight languages went from 8.8 to 2.3 seconds.
       3. ⚑ Pages tagged fr, vi or el get their names joined as entities. NWordGrams::filter finds the list in the fr_FR, vi_VN or el_GR folder. A language with no list no longer looks for its file on every phrase.
   75. ⚑ A dictionary merge no longer checks every tier file. Each tier merge synced the merged tier a second time. It then added every file of it to the dictionary's checkpoint, and that checkpoint came to list every tier file. Each open for writing read the size of each one, which took 18 to 28 seconds before each partition merge on pollett.org. Tier files are written once and synced before they take their place, so they are no longer checkpointed. An old checkpoint is dropped when the dictionary is opened. Tier files now hold up to 256KB in place of 32KB, so a merge syncs and deletes an eighth as many files. A merged tier is moved aside and deleted later, a second's worth after each file the indexer takes. Merging two tiers of 800,000 rows here took 1.15 seconds in place of 1.77. A tier merge now copies rows as the tier files store them and decodes only keys. Where both tiers hold a key, the two rows are joined without decoding their records. Merging 20,000 keys of 200 records each took 0.4 seconds in place of 0.55. The bundle's checkpoint no longer lists tier files, so fetchers stop reporting them gone. Nor does it list a built partition's files, which never grow, so a build's end no longer reads every file the checkpoint names. After a partition merge, tier 0 and at most one higher tier now merge. On pollett.org tiers 0, 1 and 2 merged at once in 28 seconds, 4, 8 and 15 each. That work is now spread over several partitions. No more than two tiers are left full, since a lookup reads both slots of a full tier. A lookup walks the tiers again when a merge's two renames fall inside its walk. A merge count kept in the tree's folder is odd between them. So a walk never gets a word's rows twice or not at all. A fetcher's slow page line now gives the page's language and length. hyphenateEntities no longer trims words split on white space. A picture more than 1,600 points on a side is scaled down before tesseract reads it. A 2,400 point wide picture read the same at half size, in 0.25 seconds against 0.33 to 0.40. In the logs of 26 October, compressing summaries was still 3.7 of the indexer's 6.6 seconds per file. So fetchers now compress each summary, as they do pages, and the indexer stores those bytes as sent. A summary still gets its url hash added before it is compressed. A sentence of more than 512 characters, as on pages with no periods, is now cut into pieces at spaces. A Bloom filter lookup that unpacked one word at a time gave no gain above noise, so it was not kept. The crawl then fell from 200,000 to 240,000 pages an hour to 160,000 to 180,000. Each fetcher still sent the whole site beside its encoded summary, so every summary was compressed and sent twice. A send carried 1,700 to 6,600 bytes a site, against 740 to 920 in that morning's logs. The site part now holds only the six fields the indexer reads. Packing 2,000 test sites took 0.30 seconds and 7.8 MB, against 0.55 seconds and 15.0 MB. With that, the crawl still ran under 180,000 pages an hour, so both changes are taken back. Fetchers again send the site whole and the indexer compresses summaries. The cap on sentence length stays. The rate had not fallen. The 200,000 to 240,000 came from the indexer working off a backlog of 1,726 fetcher files on 25 October. It held 592 at 07:41 on 26 October and none by 17:56. Each file raised the count by about 650 pages, about 70,000 an hour while the backlog lasted.
   76. ⚑ The indexer stores a fetcher's file in less time. On pollett.org it spent half of each file compressing summaries at gzip level 6, and 14% saving the partition index after each slice. Summaries are now stored at SUMMARY_COMPRESSION_LEVEL, 3, and pages stay at 6. The partition index is kept in memory over a file's slices and saved once at its end. So searches see a file's pages seconds after it arrives. Storing 60,000 pages here took 3.3 seconds in place of 5.0. A writable open of the bundle now logs its checkpoint check's time. Opening it to merge a partition took 18 to 28 seconds. A slice here is a group of up to 5,000 sites of one file. The constant is now SITES_STORED_AT_ONCE, and the comments about it say so in full.
   77. ⚑ Crawl Speed Data shows the indexer backlog and the fetchers' downloads per hour. Visited Urls/Hour counts pages as the indexer stores them. So it read 240,000 while the indexer worked off a backlog, with fetchers at 175,000. Two columns now show both.
       1. ⚑ Each queue server puts its count of fetcher files waiting in its crawl status as INDEXER_BACKLOG. aggregateStatuses sums it over the machines.
       2. ⚑ A fetcher sends its downloads per hour with each crawl time check. FetchController keeps it in the robot table, and fetcherDownloadsPerHour sums the fetchers heard from in the last ten minutes.
       3. ⚑ CrawlSpeedJob logs both in two more columns, and an older line shows them empty. On a phone the numbers stay whole and the table scrolls sideways under a fixed Time column.
       4. ⚑ The columns follow a page through the crawl: Time, Downloaded/Hour, Index File Backlog, Indexed Pages/Hour, Indexed Page Count, Total Urls Seen. The four renamed labels are new strings in all 25 locales.
       5. ⚑ Downloaded/Hour read 0 on Chris's crawl, so the fetchers' rate no longer comes from MediaUpdater reading the name server's robot table. A fetcher sends its rate with each upload of data. Each queue server keeps the rates of the fetchers heard from in the last ten minutes in its crawl status as FETCHER_RATES. aggregateStatuses keeps one rate per fetcher, the newest, and sums them as FETCHER_DOWNLOADS_PER_HOUR, which CrawlSpeedJob logs as it does the backlog.
       6. ⚑ A run on the local test system found two causes of the 0. A web server that loaded FetchController before the update keeps the old class until it restarts. That class writes no rate in the robot table. And a fetcher's first downloads per hour divided by the hours since 1970. Fetcher::ratePerHour now keeps the window of each per-hour figure the fetcher logs, and the first window starts when the fetcher was made. On the local system the first figure read 360 and the speed log read 327.
       7. ⚑ CrawlSpeed.log lines no longer hold the crawl's channel, which was 0 on every line of Chris's log. crawlSpeedHistory reads lines with and without it.
       8. ⚑ In Chris's fetcher logs of 27 October, 848 pages of vx-3.com took 3,033 of the 4,470 seconds of HTML processing, 3.6 s each, mostly the summary's scoring step. scoreSentencesVersusAverage now scores each sentence once and lowers only the scores a pick changes. The slow page line also gives how many terms the sentences hold, to show why such pages are slow.
       9. ⚑ The vx-3.com page Chris sent explains the slow summaries. The fetcher keeps the first 996,326 characters of a 1.7 MB page, which ends inside a script. removeTagSpans now takes out an element with no close tag, up to the end of the page. That script's JSON is no longer read as 1,189 sentences, and the page's summary takes 0.02 s, not 0.76 s.
       10. ⚑ A host may give a fetcher SLOW_PAGES_BEFORE_FAST slow pages in an hour. Its pages then get the fast variant of their processor until the hour ends. For HTML that is HtmlProcessor with the ScrapeSummarizer. Each fetcher keeps its own counts.
       11. ⚑ The logs of 27 October show uploads taking about 27 percent of fetcher time. Sending was 3,154 parts of 2 MB, 12 percent. Each fetcher now keeps a part size for each queue server. It starts at DEFAULT_POST_MAX_SIZE and grows to the limit the server reports. When a size fails twice in a row, the fetcher restarts the upload. It then searches between the largest size that went through and the smallest that failed.
       12. ⚑ Under Yioop's own web server, FetchController reports MAX_REQUEST_LEN less POST_HEADER_ROOM as its largest post, not PHP's post_max_size. That server does not use post_max_size. The first part of an upload now starts its file again, and the file name keeps only hash characters.
       13. ⚑ Under Yioop's own web server, index.php raises MAX_REQUEST_LEN to the Git push size limit. That limit is none by default, so fetchers were told they could post any size. FetchController now gives post_max_size, raised to the web server's default and kept under MAX_REQUEST_LEN.
       14. ⚑ ARCHIVE_COMPRESSION_LEVEL is now 4, as Chris chose. On sample pages it takes 0.293 ms a page against 0.532 ms at level 6, with output 2.4% larger.
       15. ⚑ index.php no longer raises the web server's request limit to the Git push size limit, as Chris asked. The limit follows post_max_size only, raised to the web server's default. GIT_MAX_PUSH_SIZE stays a cap of its own on git pushes under that limit.
       16. ⚑ GIT_MAX_PUSH_SIZE is gone, as Chris asked; a git push is held to post_max_size. Its field on the users, roles and groups page and its locale string are gone too.
       17. ⚑ Profile.php holds only fields a page lets an administrator change. GROUP_ITEM and the four PRIVATE_DB fields left the list and come from Config.php. Database version 137 writes Profile.php again without them.
       18. ⚑ Chris's crawl kept reading 0 after every machine restarted. His error log of 15 October shows the robot table write failing with no space left on the device. That write emptied the table, and FetchController never wrote an unreadable table again. The queue server then read nothing on every pass. FetchController::recordRobot now starts such a table again. It changes the table under an exclusive lock and makes it readable by the crawl processes. The queue server reads it under a shared lock.
       19. ⚑ SmtpClient::smtpCommand notes in its log a command written to a closed connection. The notice it raised before showed up in a suite run under load.
3. ▶ An index is never left corrupt by a crawl that stops part way through. Phase 4 added guards that keep a corrupt index from taking the server down. Those are guards rather than prevention. This item is the prevention.
   1. ⚑ Every point that could leave a half-written entry is named. The walk covers the partition append, the dictionary merge and the postings write. What is looked for is a write that a later read takes as a garbage length or offset. Nine were found.
      1. ⚑ PartitionDocumentBundle::put and advanceSavePartition save the partition index in place. A kill part way leaves offsets and lengths that point at garbage.
      2. ⚑ PartitionDocumentBundle::addArchive appends each document to the partition file. A kill part way leaves a partial tail that no index entry names yet.
      3. ⚑ IndexDocumentBundle::buildInvertedIndexPartition writes doc_map, postings, last_entries and positions in place, one after another. A kill part way leaves a mix.
      4. ⚑ mergeNextPartition checks only that those four files exist, so it can merge a truncated postings file into the dictionary.
      5. ⚑ IndexDocumentBundle::updateDictionary writes next_partition.txt in place. A kill part way leaves it empty, and it reads as partition 0.
      6. ⚑ LSMTree::flush and writeRecords write tier data files and file lists in place. A kill part way leaves a record list with a cut last line.
      7. ⚑ LSMTree::mergeTier writes the merged tier while both source tiers still exist. A kill part way leaves a partial target beside complete sources.
      8. ⚑ LSMTree::getMaxTier and IndexDocumentBundle::setArchiveInfo write small files in place; the parameter files alone go through a rename.
      9. ⚑ The build, merge and slot claims are opened with x, so a kill leaves only a stale claim, which the timeout already clears.
   2. ⚑ An undo log guards each write point named in A. Before a step changes any index file, it records how to undo the change. When the index next opens, any step left unfinished is undone.
      1. ⚑ The log is one file in the bundle folder. A step appends a record naming itself and each file it will change, then syncs the log.
      2. ⚑ For a file the step appends to, as the partition file and positions are, the record keeps the file's length before the step.
      3. ⚑ For a file the step rewrites in place, the record keeps a copy of the old file, taken before the first change.
      4. ⚑ When the step has written and synced all its files, it appends a record saying it finished.
      5. ⚑ On opening a bundle, each step with no finished record is undone. Appended files are cut back to their old length, and rewritten files are copied back.
      6. ⚑ An undone partition build or merge is then run again, as it would be if it had never started. A log line says what was undone.
      7. ⚑ Small files that are written whole, such as next_partition.txt and max_tier, are instead written to a new name and renamed.
      8. ⚑ Once every step in the log has finished, the log and its copies of old files are removed.
      9. ⚑ UndoLog does this. A partition build, a partition's dictionary merge with the partition count, and each tier merge are steps.
      10. ⚑ A step whose process still runs is never undone, so a search opening the index during a crawl leaves a merge alone.
      11. ⚑ A tier merge writes under a name searches never read, renames it in, and commits. A later stop then completes the removal.
      12. ⚑ A dictionary read now also reads each tier's B slot, and finds the highest tier again when the one it held is gone.
   3. ⚑ The recovery is shown by stopping a crawl on purpose. A synthetic crawl is driven and the queue server is stopped at chosen points in the write path. The index is read back afterward and must come back clean every time.
      1. ⚑ UndoLogTest kills a build, a dictionary merge and a tier merge at 0, 2 and 10 milliseconds into each step.
      2. ⚑ Each time the bundle opens whole, and holds every merged partition exactly once after the step runs again.
      3. ⚑ Each stop case now removes its bundle when done, so the suite leaves no folder under tests/test_files.
   4. ⚑ Two processes never build the same partition at once. A claim now holds its maker's process id and stands while that process runs.
      1. ⚑ On a synthetic crawl a partition of about 147,000 documents took 12 to 13 minutes to build, past the 10 minute timeout.
      2. ⚑ So claims and build slots looked abandoned mid build. Six of seven fetchers built at once, two of them partition 32.
      3. ⚑ IndexDocumentBundle::claimIsAbandoned now decides, for the merge claim, the build slots and the partition claims.
      4. ⚑ Builds slowed about 29% with combined terms, not ticks. Last rows are now plain arrays until saved, 13% faster, same files.
      5. ⚑ A word's term ids are made once per word and page combination and reused, about 6% faster, with the same files.
      6. ⚑ A term's postings are appended in place and counted once at the end. Merging rows had cost time in the square of a row.
   5. ⚑ Crawls survive stops, power cuts and races between processes. A partition's files count as built only when they hold its documents.
      1. ⚑ The undo log syncs each write and keeps a checkpoint, so a power cut rolls a file back to it.
      2. ⚑ A crawl stopped before saving a document still stops. A partition read as empty is not built as empty.
      3. ⚑ Each queue server process reads a start message once, under a lock.
      4. ⚑ A partition built while it was still being filled is built again before it is merged. A built_from file holds the hash of the document index the build read. In a local crawl with 100 documents a partition, partition 0 had gone into the dictionary with 1 of its 99 documents.
   6. ⚑ The news feed index stays whole when MediaUpdater is stopped. A feed partition's files are saved as one undo log step, and every item is indexed. A saved second seen-item filter no longer stops the bundle opening. A feed partition's terms are saved in order, so the dictionary finds them; ArcTool rebuild IndexDataFeed mends a bundle saved before. An update that adds a partition still saves the seen-item filters, so the next update does not add its items again.
   7. ⚑ A partition not added to the dictionary is not recorded as merged. On yioop.com, info:https://www.sjsu.edu/ and info:https://www.sjsu.edu/robots.txt found no dictionary row. The site was a seed, and robots.txt pages are indexed with their meta words.
      1. ⚑ Here, the query's key for each url is the key a build files the url's info: word under. On five local indexes, info: queries found their page.
      2. ⚑ updateDictionary ignored a false return from addPartitionPostingsDictionary, which comes when the partition's postings or counts file is missing. It recorded the partition as merged anyway, so none of its words could be found again.
      3. ⚑ The partition now stays the next to merge, and the log says so. Whether this is what dropped the San Jose State University rows on yioop.com is not shown.
   8. ⚑ A rebuild says where it can safely continue from. A rebuild printed every DictionaryUpdater log line and a progress line every two seconds. It now keeps those logs in files and prints its state only when it changes, naming the safe partition to continue from. The new rebuild-status command prints the same for a stopped rebuild.
4. Other work, grouped by what it touched. Each is a deliverable of its own that came up during the arc. They are grouped by the part of Yioop they concern rather than listed one by one.
   1. ✓ The unit tests pass with no warnings or log lines. Each case cleans up after itself and needs nothing installed.
      1. ⚑ cullByDomainFilterTestCase makes its own filter instead of needing a top ten million domain file.
      2. ⚑ WebSiteTest puts back the request globals, so the runner's NO_LOGGING setting survives. WebSite reads a missing SCRIPT_NAME as the site's root.
      3. ⚑ PartitionDocumentBundleTest removes the folders it makes, and LSMTree checks for a renamed file instead of reading it.
      4. ⚑ TurnServer.php leaves NO_LOGGING alone when a test only reads its class, so later tests no longer print indexer log lines.
      5. ⚑ yioop_error_handler leaves an error silenced with @ to PHP, so it is no longer printed.
      6. ⚑ A mergeNextPartition case covers a partition not yet built, a fresh and an abandoned merge claim, and the merged dictionary.
   2. ✓ The mail library no longer translates. Backends throw a plain Exception starting with an ImapClient or MailBackend ERROR_ constant; MailComponent::mailErrorText matches the constant.
      1. ⚑ MailBackendException is removed. MailCloneJob asks ImapMailBackend::isTransientError whether to try again.
      2. ⚑ Every string MailComponent uses begins with mail_component_, including eight that had never been added to any locale.
      3. ⚑ The api_unsubscribe strings are unsubscribe_view strings; the other api_ strings, used nowhere, are removed.
   3. ✓ Serve with several web server processes. The web server keeps every connection, and runs requests in worker processes when Web Server Workers is above zero.
      1. ⚑ WebSite starts the workers with proc_open once it runs as its run-as user, and links them by loopback sockets.
      2. ⚑ Each request goes to the least busy worker with its globals and session records, and the reply comes back with any changed sessions.
      3. ⚑ Requests whose replies stream, such as resources and byte ranges, stay in the web server, as WebSocket connections do.
      4. ⚑ Workers share the file write journal, and exit when the web server stops. With one process nothing changes.
      5. ⚑ A WebSiteTest case starts a site with two workers and checks replies, sessions, a streamed reply and worker exit.
      6. ⚑ The server's select also watches the worker links, so a reply wakes it. Yioop keeps /wd/ paths in the server, as resources.
      7. ⚑ Each access line for a relayed request ends with the number of the worker that ran it.
      8. ⚑ A restart no longer signs users out. The supervisor hands the new server the sessions saved in the last five minutes.
      9. ⚑ WebSite::loadSavedSessions takes them back; before, nothing read them, and the restart page's first status poll got a new session.
      10. ⚑ A worker takes the front's server settings with each request. Before, a request it made to its own site, as Manage Machines does to start a queue server, went to the error page.
      11. ⚑ A worker link is accepted only after the worker shows a hash of a secret the front gave it. The front answers with its own hash.
      12. ⚑ Connections to the worker link are read side by side, so silent ones cannot hold up a worker's start. Each is closed after half a second.
      13. ⚑ Refused connections are noted in the worker log, five a minute and then a count. A worker that fails to start waits 30 seconds.
      14. ⚑ A worker that stops is replaced on the next request. The worker log records each start, failure to start, and stop.
      15. ⚑ A worker that stops is replaced at the next request. Each start, failed start and stop is written to the worker log.
   4. ✓ WebSite can share a file write journal between processes. Each process forgets a cached file another has written, so it neither serves the old copy nor skips a needed write.
      1. ⚑ Two server objects on one file without it: a stale read and a lost write. With it: neither.
   5. ✓ Yioop no longer uses pcntl, except to start programs. Its signal and alarm uses become tick functions, with ticks declared every 1,000 statements in the files they must reach.
      1. ⚑ runWithTimeLimit stops a document's slow parsing through ticks in PhraseParser and the 25 tokenizers. It adds 6 percent to phrase extraction.
      2. ⚑ MediaUpdater checks for a stop once a second, even inside a job. MailSite's watchdog reports a stuck event loop once.
      3. ⚑ ArcTool's rebuild and remerge run in a process of their own, so closing the terminal does not stop them.
      4. ⚑ MailServer's signal log is dropped. CrawlDaemon still forks to start a program, since otherwise it keeps the web server's port.
   6. ✓ The Web Server legend has a Log link. It shows beside the help button only when Yioop is its own web server, and reads Log in each locale.
   7. ✓ The web server reports how busy it is, and its log opens from the Web Server settings. Each memory sample line now gives the server process's processor use since the last line. A value near 100 means one core is full.
      1. ⚑ Sampled directly, the figure read 0.0 after two idle seconds and 99.9 after two seconds of busy work.
      2. ⚑ The Web Server fieldset links to index.log through the machines log view. The link returned the file's lines.
      3. ⚑ The link has its own string, servers_element_web_server_log_view, in all 25 locales.
   8. ✓ Form labels draw their markup, and the keyword captcha previews. Bold in a text field's label showed its quote marks in both parsers. The editor's preview did not draw a keyword captcha at all.
      1. ⚑ Both parsers now draw markup in the label of text fields, text areas, checkboxes, radio buttons and the keyword captcha.
      2. ⚑ The preview draws the captcha's label and box without its answer. New cases fail on the old code and pass now.
   9. ✓ A large file downloaded from the web server does not stall other requests and arrives whole. One reader downloading a large video held up every other request for the length of the download, and a three hundred megabyte video ended the web server process. A whole file request is now handed to the event loop, which sends the file a block at a time between other requests, and a ten megabyte pdf that had come back as headers with no body now arrives complete over both HTTP/1.1 and HTTP/2. Measured: a page answered in twenty seconds during a two hundred megabyte download before and under five milliseconds after.
   10. ✓ A document thumb on a wiki page shows a picture, not a frame. A resource-thumb mark for a pdf drew an iframe of the document where no thumbnail was there. The mark is now always a link with a picture, the thumbnail is made from the document on the first read that asks for it, and an image thumb draws the thumbnail address rather than the full picture.
   11. ✓ A reader who may not edit, or see a private history or source, is answered 403. A stranger's arg=edit with a stale token was quietly downgraded to a page read and served 200; a private page's source was served at 200 too. The group controller now answers 403 for a writing arg on a request whose token is bad; the wiki component refuses edit, history and source by the group's Access setting, where only Read Write Wiki lets a member edit. Walked as a stranger, a member of a Read group, a member of a Read Write Wiki group, and root.
   12. ✓ The server log names the signed in user, marks each reply's speed, and names a timer that held the loop. Each finished-reply line carries the username after the address, and its time ends in -slow, -slower or -slowest by band so one grep finds a band. A timer callback that runs past a tenth of a second writes a line with its length and the file and line it was set at, since a request arriving during one waits the whole of it and two cheap 403s that took 2.6 seconds in the same second had waited on one such callback.
   13. ✓ Every request's completion line in the server log says how long it took. The line that records a reply's status and size ran only for an immediate HTTP/1 reply, so no HTTP/2 page and no deferred request, which is every search, ever logged one. It now runs once on every reply path, a deferred reply logs after its own environment is applied with a file reply's true size, a redirect logs as 302, and the line ends with the milliseconds since the request arrived. Measured here: a search over h2 22ms, the same page over h1 3ms, a ten megabyte download 2ms to hand off.
   14. ✓ A web server that has given up root can still start, stop and restart what needs a privileged port. The server binds 80 and 443 as root, then becomes the user WWW_USER names. After that it could not start the mail server, nor a replacement for itself. A restart could leave the site down with nothing in any log. The process that started the server stays as root and starts what is missing.
       1. ✓ One process keeps root and starts what is missing. It is index.php in a supervisor mode, begun by the start once the server is on its way. It binds nothing and listens to nobody, so it takes no request and needs no channel. Every few seconds it compares what should be running against what is. What should be running it reads where that was already written: the rows Manage Machines keeps and the address the server records. It adds no file of its own. It lives through a restart and ends on a stop.
       2. ✓ A stop applies only to services that were there when it was asked for. The old server and the launcher acted on the same services with no ordering. A late stop could then end a mail server already brought back. A stop now carries the moment it was asked for and leaves a younger service alone. A lock file carries the time its service began, so the two can be told apart.
       3. ✓ Every silent death now writes down its reason. The mail server's bind results, and a failure before it can listen, go to its log rather than to output that is thrown away. Every place that removes a service's lock names the process, the reason and the calls that led there. A service says how long it ran when it stops.
       4. ✓ A restart from the command line reports for a reader who barely knows sudo. It ends with which services are running again and which are still starting. For one that did not come back it gives the command to check and the path of its log. It waits for nothing. A restart from the Servers screen brings the site and its services back the same way.
   15. ✓ A scraper that rotates its address is turned away by what it asks for. A count kept per address never slows a scraper whose every request comes from a new address. The count is kept on the thing it wants instead.
       1. ✓ A wiki page whose history or source draws many refusals is refused before any work. Twenty refusals in five minutes make a page hot for an hour. A reader with no session then gets a bare 403. The web server of index.php checks before the session. Apache checks a marker file under work_directory/hot.
       2. ✓ The cached copies of an index are guarded the same way. In a five minute log from yioop.com, 1,058 cache requests came from 1,040 addresses. 1,046 were misses, each walking every index, at 878 ms on average. A miss by a reader with no session is now a strike against the index the request names. The busiest single page drew 19 requests, so the key is the index, not the page. The walk ran on a server begun with secure-start. The twentieth miss made the index hot. Then came 403 without a cookie, 302 with one, and 200 for an icon.
       3. ✓ With Cache unchecked, a cached copy answers 404. The Page Options Cache box only hid the link, and the copies were still served. The cache activity now answers 404 when it is off. A search result's icon names a field and is still served. Checked after a secure-restart with Cache off, then ended with secure-stop.
       4. ✓ A cache miss no longer prints stray words before the page. ParallelModel echoed "very null" ahead of the body on every miss. The echo is removed.
   16. ⚑ A search for a site's name finds the site first. Relevance is one divergence score over weighted fields, with url signals as pseudo words.
       1. ⚑ The shortest url of pages with the same content keeps their words.
       2. ⚑ A site's name typed as words also looks up the joined host word, and keeps the words together. A host or path holding the joined words gets 0.9 of its keyword bonus.
       3. ⚑ A union of query parts is term at a time with accumulator pruning, after Lester et al. A part with a joined word may open ten accumulators, so tik tok finds tiktok.com.
       4. ⚑ A search no longer runs the index's undo, which held a worker on every file of the checkpoint.
   17. ⚑ Searches use the index as it is now, and the web server's workers keep up under load. A worker never holds a request for good, and the crawl keeps a worker of its own.
       1. ⚑ A web server process reopens an index another process changed. It checks the index folder at most once a second.
       2. ⚑ A new default index and new debug settings apply without a restart.
       3. ⚑ A search over queue servers stops once every document is asked for, and the offset a server takes is capped.
       4. ⚑ A worker treats the name server's host as its own. A search then runs its queue server request inside the worker.
       5. ⚑ The front ends a worker whose GET has run five seconds from when it took it up. A silent worker is ended after thirty.
       6. ⚑ The crawl's requests have the last worker, which takes other requests while it is idle.
       7. ⚑ A request for a hot target with no session is refused by the front. Strikes are counted under the hot record's lock.
       8. ⚑ Requests reach a worker from a list written without waiting. A depth-one git clone through two workers fell from 51 to 5.5 seconds.
       9. ⚑ A full git clone of many versions of large locale files builds its pack ten times faster. A delta copies the shared start and end of two versions whole and compares the rest in pieces. A chain runs up to 50 deltas.
       10. ⚑ A delta is built only on an object of its own type. A commit built on a tag reached git as a tag, which stopped the clone with "invalid tag".
       11. ⚑ Workers stop through CrawlDaemon::stopProcess on Linux, macOS and Windows, and log the calls they were in.
       12. ⚑ A fetcher sends what it found after its first hundred downloads. The downloads between sends then double, up to 1,000.
       13. ⚑ The summarizer and term frequencies count a page's words once. Image processors shrink a sample of a large picture.
       14. ⚑ Partition builds and the crawl queue sync less. signalShares reads only next_partition.txt.
       15. ⚑ A secure-terminal also starts the root supervisor, which stops when the terminal does.
       16. ⚑ A company domain's root page counts the host and domain boost words, so www.yahoo.com ranks above sports.yahoo.com.
   18. ⚑ Tests start helper processes the same way on every system. UnitTest::startScript and CrawlDaemon::endProcess replace proc_open in the test cases. A case runs a helper method of its own class with UnitTest::startMethod, so tests/test_files holds no PHP script. A profile written to another folder no longer changes the running process's settings, so the full run completes. CodeTool prints the error and test that end a run. The undo cases wait for a killed process to end, then open the bundle for writing, which runs the undo.
   19. ⚑ The web server keeps its workers running. A request starts any stopped worker again, and the front sleeps on a worker's link while it waits.
       1. ⚑ The supervisor, the workers and the mail sender start as php through CrawlDaemon. Workers hold none of the front's listening sockets.
       2. ⚑ A stop of the whole site also stops what Manage Machines started. The supervisor names any process holding a port it needs.
       3. ⚑ A stopped worker is logged with the request it was running, and the memory report names each worker's request.
       4. ⚑ A request waiting behind a worker that is ended goes to another worker. Only the running request gets a 502.
   20. ⚑ Mail one server cannot take yet is tried again for days, and reaches a domain's hosted server. Other recipients still get it. A domain with no MX record is tried at its Microsoft 365 or Google Workspace server.
   21. ⚑ Yioop runs under Apache and PHP-FPM without notices or clashes. A request no longer writes Profile.php, and one request at a time runs an upgrade. Web logging goes to index.log, and style sheets keep their own headers.
   22. ⚑ The private database always has a name, and a person using the site is not refused. A signed-in person with an old token is sent back with a fresh one.
   23. ⚑ Outside programs are used once installed, with no setting needed. Yioop asks the system where ffmpeg or tesseract is. It uses where on Windows and command -v elsewhere, and skips the work when neither finds the program.
       1. ⚑ A setting such as FFMPEG in LocalConfig.php still wins. A setting of false turns the work off.
       2. ⚑ WHISPER and the three transcription settings, which no code read, are removed.
       3. ⚑ Temporary files for git packs, zip downloads, certificate requests and node tests go under C\TEMP_DIR.
       4. ⚑ On Windows the memory size comes from PowerShell's Get-CimInstance, since wmic is not installed on newer versions.
   24. ⚑ A saved form no longer sends the browser to 0.0.0.0. A redirect to the site's base gives only its path, so the browser keeps its host. baseUrl does not use 0.0.0.0 or :: as a host name.
   25. ⚑ A program ended with Ctrl-C on Windows no longer blocks its next start. isProcessAlive looks the id up in what tasklist lists, read at most once a second. Before, Windows had no check, so every lock looked held.
   26. ⚑ A connection that closes before its TLS handshake no longer keeps the web server busy. The front's loop spun on such a socket for up to 30 seconds. The [website-front] line now counts them as closed_before_first_byte.
   27. ⚑ Each bundle of packed objects is cut by its own dictionary. A small bundle right after another was cut by the earlier one's numbers. A magazine's page tree then came apart, and a later page was drawn as its cover.
   28. ⚑ A request over HTTP/3 keeps the server's script name. Each HTTP/3 request named an empty script name, so Yioop sent every path to the error page. The web server now also logs a client's stream resets and closing errors on HTTP/2.
   29. ⚑ An HTTP/2 frame of an unknown type is skipped. A PRIORITY_UPDATE frame ended the whole connection with an exception, and every response still owed on it was lost. Such a frame is now read by its length and passed over, as RFC 9113 asks.
   30. ⚑ A remade thumbnail is served whole. The web server kept the old picture in its cache of small files. It sent those bytes under the new picture's length until it was restarted. The remade picture is now written through the web server's journal, and the length sent is that of the bytes sent.
   31. ⚑ An exact match of a typed word doubles its share of a result's relevance. The index holds stems, so anime also finds animal. Once a result's title and summary are in hand, each typed word found in them multiplies its share by EXACT_MATCH_FACTOR.
   32. ⚑ The guards from the tenth-round review are in the process file and its scripts. A withdraw now counts the round, and a cut prints the git add line of each new file. The word check warns about an acronym not spelled out and a missing To check line.
   33. ⚑ Manage Users shows when each user joined. The list of users gains a Join Date column, and the edit form a Join Date row that cannot be changed. Both read the account's CREATION_TIME. The new locale key takes each locale's wording of the group join date.
   34. ⚑ Errors in yioop.com's message log are fixed. The log held a Postgres error from the wiki page list and thousands of false checkpoint notes. It also showed keys reported absent during tier merges.
       1. ⚑ WikiModel::getPageList no longer uses SELECT DISTINCT, which Postgres refuses with ORDER BY LOWER(GP.TITLE). A category is matched with IN, so each page still comes back once.
       2. ⚑ A bundle's undo checkpoint drops the tier files a dictionary merge removes, and the queue server's removed partition marker. They were reported as lost each time the bundle opened.
       3. ⚑ LSMTree::emptyTier moves a tier aside in one rename before deleting it. A lookup during the deletion then sees the tier gone.
   35. ⚑ A web server worker writes its log lines to index.log. A worker never named its log, so lines it wrote during requests went to message.log.
   36. ⚑ A word's walk no longer skips the last posting of a partition. In a one-word query, the field key's score was lost for each partition's last page with the word in host, title or path. That page then ranked as if the word were only in its body.
       1. ⚑ WordIterator::advanceSeenDocs counted the postings left from next_offset, which findDocsWithWord had already moved. It now counts from current_offset, where the block starts.
       2. ⚑ getPostingsSliceResults took a descending block of at least results_per_block. It now takes at most the postings left, so a walk from last to first reads no page twice.
   37. ⚑ A recrawled page is counted once in a word's union of keys. A page crawled twice has a posting for each copy in a word's own key and in its field key.
       1. ⚑ WordIterator gives both postings the latest copy's key, and UnionIterator added the copies' rank and relevance. The page then scored about twice its relevance.
       2. ⚑ UnionIterator now keeps the higher scoring copy when two documents share a key. A link's score is still added to the page it points to.
       3. ⚑ In a scratch index with a song page crawled twice, the song page's relevance for yahoo went from 24.07 to 12.03. yahoo.com, at 12.44, now ranks first.
   38. ⚑ A one-letter query is looked up as typed. guessSemantics rewrote a query of one letter, such as x, as letter x|x.
       1. ⚑ The added part matched pages holding both letter and x, so dictionary pages about the letter x led the results for x.
       2. ⚑ The rewrite is removed. In a scratch index, x.com went from second to first for x.
   39. ⚑ Results much like one above them move down. PhraseModel::diversify reorders the top three pages of results by maximal marginal relevance, after Carbonell and Goldstein (1998).
       1. ⚑ Each next result has the highest lambda times its share of the top score, less a penalty. The penalty is 1 - lambda times its similarity to a result placed.
       2. ⚑ Similarity is the cosine of the title word sets, without the query's words. Shared words are counted once from per-word lists of results. Nothing in the index changes.
       3. ⚑ Every page inside the window is cut from one reordered list, since getSummariesByHash fetches the same summaries for each of them.
       4. ⚑ Lambda is the profile value MMR_LAMBDA, 0.7 by default, set on Page Options. 1 keeps the order by score.
       5. ⚑ On yioop.com's results for x, x.com went from 5th to 3rd. The second and third dictionary.com pages went to 8th and 9th.
   40. ⚑ Searches no longer stop on a missing MMR_LAMBDA. MMR_LAMBDA was a profile setting defined only for a site with no profile.
       1. ⚑ On a site whose profile predates it, every search threw "Profile constant MMR_LAMBDA not defined".
       2. ⚑ The weight is now MMR_RELEVANCE_WEIGHT, a plain Config.php setting. The profile field, Page Options field and locale string are gone.
       3. ⚑ The new name keeps a leftover MMR_LAMBDA line in a saved profile from being read.
   41. ⚑ A term in the active partition is found by binary search. getWordInfo reads the postings of the partition still being filled as one string, and searched the whole string for each term.
       1. ⚑ A query looks up each page it scores this way, to find the page's latest copy. Each lookup read the whole string.
       2. ⚑ PackedTableTools::findRowInSortedTableString does a binary search on byte positions. The rows are saved sorted by term.
       3. ⚑ On a scratch index with 39,000 pages in its active partition, y combinator went from 5.7 seconds to 0.4 seconds.
       4. ⚑ Of 2,242 searches from the site's logs replayed on that index, 100 timed out at 5 seconds before and 2 after.
   42. ⚑ A search with a negated word gives every page without it, and quickly. Two searches from the site's logs with -eliminate -abuses took over 5 seconds on the scratch index.
       1. ⚑ DocIterator read a partition's whole document map again on every reset. One such search made 52 walks, each reading it three times.
       2. ⚑ A walk now reads rows through the bundle's doc_map_tools, which keeps a map and its row positions for every walk of a process.
       3. ⚑ A place in the walk is a document map row number, the DOC_MAP_INDEX a WordIterator gives for the same page.
       4. ⚑ A walk stopped before each partition's last row, so ten pages in partitions of four gave seven. Walks in both directions now give ten.
       5. ⚑ NegationIterator stopped once the negated word had no page left. Without rare, on one page of ten, it gave two pages; it now gives nine.
       6. ⚑ The slower search went from 5.2 seconds to 0.23 seconds. In a replay of 2,242 logged searches, none timed out, against one before.
   43. ⚑ Search suggestions sit under the search box and list the most frequent completions. On a page of results, the dropdown was fixed 50 pixels from the top and began 9 pixels inside the search box.
       1. ⚑ placeDropdown sets its top, left and width from the box when the page fixes it on the screen. The landing page is unchanged.
       2. ⚑ It lists the six completions of highest frequency, after matching history. For la it made 382 rows before.
       3. ⚑ For en-US, completions under the capitalized prefix are found too, so eng gives England.
       4. ⚑ spellCheck read its history count from a variable out of scope, so a query sent six times still got a correction.
       5. ⚑ suggest.js's docblocks are brought up to standard.
   44. ⚑ yioopbar.xml searches yioop.com over https. Its search URL template began http://www.yioop.com.
   45. ⚑ A search reads each dictionary data file once. Each page a search scores looks up its newest copy under an info: key. That walk read a whole data file in every tier, though most pages share the files of the small tiers.
       1. ⚑ Tier::get keeps a data file's entries in the cache that holds the index lists, checked against the file on each read.
       2. ⚑ It finds the key by binary search over the sorted entries. The 8 million entries of two scratch indexes held no key twice in a file.
       3. ⚑ For mary pollett over 300 partitions, a search read 927 data files totalling 30.3 megabytes, but only 116 different ones. It now reads 121.
       4. ⚑ lookupLatestVersionPage asks the dictionary itself rather than making a model, which connected to both databases for each page.
       5. ⚑ IndexManager.php's docblocks are brought up to standard.
   46. ⚑ QueryTool times three parts of opening a partition for scoring apart. yioop.com's mary pollett run spent 1.06 s opening partitions for scoring. Positions file opens, document map loads and signal share reads now print as lines of their own.
   47. ⚑ A search scores the newest copy of a page wherever it is. A page's newest copy was looked for only in the active partition, so a newer copy merged into the dictionary was never found.
       1. ⚑ A lookup read the postings of the newest row only when that row was the active partition's. A dictionary row holds only where its postings are.
       2. ⚑ IndexDocumentBundle::newestRow asks the active partition first, then walks the dictionary from its lowest, newest tier and stops at the first row.
       3. ⚑ LSMTree::get takes $newest_first for this walk.
       4. ⚑ A newer copy without the query's word drops the page. One with it is scored from its own partition's document map and positions.
       5. ⚑ checkTermExists tested bit $hash % 7 where storeTerms set $hash & 7, so every newer copy found was taken to lack the word.
       6. ⚑ The terms filter holds plain words, so it is used only for a plain word's key.
       7. ⚑ Lookups found are remembered, so mary pollett on a 300 partition test index made 200 lookups, not 601.
       8. ⚑ A word iterator asks for the word's whole count only with its first batch of rows, the only batch whose totals are read.
       9. ⚑ The Crawl Speed History strings are translated into every locale.
       10. ⚑ IndexDocumentBundle.php's docblocks are brought up to standard.
   48. ⚑ A search reads one document map entry without reading the whole map. Scoring a posting loaded its partition's whole document map, about 3 MB on yioop.com, to read one entry.
       1. ⚑ Each partition gets a doc_map_marks file: the map's size, its entry count, and where every 32nd entry starts.
       2. ⚑ IndexDocumentBundle::docMapEntry seeks to the mark before an entry and reads 4 KB at a time past at most 31 entries.
       3. ⚑ A build writes the file; an older partition gets it on its first search, and again if the map's size changed.
       4. ⚑ For 30 entries of a 3.2 MB map, the whole map read took 0.064 s and the marked reads 0.0011 s.
       5. ⚑ DOC_MAP_MARK_EVERY and DOC_MAP_READ are new in Config.php.
   49. ⚑ A search looks up the newest copy only for pages that reach the results. yioop.com's mary pollett run made 2,277 latest copy lookups, 2.224 s, for 197 result rows.
       1. ⚑ WordIterator no longer looks up a newest copy for each posting; it marks a text page's posting with TERM_KEY, the word found.
       2. ⚑ GroupIterator::newestCopy checks each copy of a grouped page once, with one lookup.
       3. ⚑ A newest copy without the word drops the copy; one with it is shown, keeping the scored copy's score.
       4. ⚑ IndexManager::termPostingInPartition, moved from WordIterator, checks the newest copy for the word.
       5. ⚑ Only the top LATEST_COPY_FRACTION, a tenth, of a block's pages by doc rank are checked, since a newer copy does not change doc rank. On the test index, mary pollett made 20 lookups, not 200.
       6. ⚑ QueryTool also times making the query iterators, scoring and grouping the results, and reading their summaries.
   50. ⚑ A search keeps each document map and positions file open while it scores a partition. docMapEntry opened and closed a partition's document map for every entry; yioop.com's mary pollett read 2,919 entries.
       1. ⚑ IndexDocumentBundle::$doc_map_handles keeps a handle per map, with the size it had, and reopens a map saved again since.
       2. ⚑ 30 entries of a 3.2 MB map took 0.0007 s, against 0.0011 s.
       3. ⚑ WordIterator keeps the scored partition's positions file and document count across calls, and reads the signal shares once a query.
       4. ⚑ On the test index, mary pollett read the signal shares 4 times, not 601, in 0.004 s against 0.075 s.
       5. ⚑ docMapEntry keeps each entry it reads for the query, so the parts of a union scoring the same page read its entry once.
   51. ⚑ Fetchers compress the pages they send, so the indexer stores them as they come. In yioop.com's indexer log, compressing archive items took 101 of 163 s, and the indexer was 1,172 files behind.
       1. ⚑ Fetcher::compressAndUnsetSeenUrls packs a text page as a PRECOMPRESSED_PAGE array holding the page serialized and compressed with gzencode at ARCHIVE_COMPRESSION_LEVEL.
       2. ⚑ PartitionDocumentBundle::put stores those bytes as they are when the bundle compresses with GzipCompressor, as an index bundle does. For any other compressor it unpacks the page first.
       3. ⚑ A page sent plain, from a fetcher not yet updated, is compressed by the indexer as before.
       4. ⚑ On 400 pages, the indexer's put took 0.016 s against 0.75 s, and the archive file was 4,621,377 bytes both ways.
   52. ⚑ A search keeps at most DOC_MAP_HANDLES_KEPT document maps open. San jose state university ran past the web server's time limit on yioop.com. Keeping up to 1,000 maps open in each process had raised the files a test index search held from 111 to 308.
       1. ⚑ docMapEntry keeps at most DOC_MAP_HANDLES_KEPT, 32, maps open and closes the one used longest ago.
       2. ⚑ WordIterator closes the positions file of the partition it leaves. The same searches now hold 143 files and give the same rows.
       3. ⚑ The document map caches drop their oldest entry with unset rather than array_shift, 4 against 22 microseconds an entry at 10,000 entries.
       4. ⚑ GroupIterator::getDirection asks the iterator below it, and currentGenDocOffsetWithWord returns its answer. GroupIterator's docblocks meet the standard.
   53. ⚑ A search checks filtered hosts in memory and looks up the index once per partition. San jose state university spent 1.910 s making its query iterators on yioop.com. A profile of the state university on a test index put 23% of word scoring in SearchverticalsModel::isFiltered.
       1. ⚑ isFiltered queried GROUP_ITEM for each new host, and forgot all hosts past 400. filteredPairs reads all filtered hosts in one query, up to MAX_SEARCH_FILTERS_LOADED.
       2. ⚑ The pairs are read again after an edit, or when lastChange gives a newer time.
       3. ⚑ WordIterator keeps the index with the scored partition, rather than calling IndexManager::getIndex for each posting.
       4. ⚑ On that query, making the iterators took 0.074 s against 0.097 s, the middle of five runs, with the same 56 rows.
       5. ⚑ SearchverticalsModel called addGroupItem, which is FeedModel's, so filtering or editing a result stopped with a fatal error. It now adds them through FeedModel.
   54. ⚑ A word iterator reads up to 500 dictionary rows in one lookup. San jose state university made 466 term dictionary lookups taking 2.2 s of the 2.9 s spent making its query iterators on yioop.com.
       1. ⚑ LSMTree::get reads a word's rows from the start of its list to the last row asked for. Read 20 rows at a time, a word's early rows were read again for every batch.
       2. ⚑ NUM_DISTINCT_GENERATIONS goes from 20 to 500, kept in each iterator's generation_batch_size.
       3. ⚑ A later batch that fills up no longer looks up the active partition's row, which it neither adds nor counts.
       4. ⚑ On a 300 partition test index, a ten word query made 20 lookups against 38, with the same rows. batchesReachEveryPartitionTestCase still walks batches of 20 rows.
       5. ⚑ A batch holding the active partition's row is the word's last. Where that row filled the batch, the walk loaded one more batch, got the row again, and read the active partition's pages twice. batchesReachEveryPartitionTestCase now walks batches of 2 rows over 4 partitions of 2 pages.
   55. ⚑ QueryTool says where a stopped query was, and a union part that does not advance steps on. On yioop.com, querytool "cs dept san jose state university" never finished, even for its plan. Here it took 0.08 s on three indexes.
       1. ⚑ Ctrl-C now prints the calls the query was in and its timings so far. Windows has no such signal, so there the tool ends as before.
       2. ⚑ A union part whose document comes before an accumulator's is advanced to it. If the advance left it where it was, the union asked again forever.
       3. ⚑ Such a part now steps on one document, and QueryTool counts it as "union parts that did not advance".
   56. ⚑ The suggested query keeps numbers and short words, and shows the query as text. On yioop.com, "pandemic legacy rule vaccinate 6 cities" was offered as "pandemic legacy rule vaccinate p cities".
       1. ⚑ correctSpelling in suggest.js changed any word not in the dictionary to its most frequent word one edit away. Every one letter string is one edit from 6, so 6 became p.
       2. ⚑ A word with a digit, or shorter than MIN_CORRECTED_LENGTH, three characters, is now kept as typed. A candidate not in the dictionary no longer takes the frequency of the candidate before it.
       3. ⚑ spellCheck put the query into the page as markup and into its link without encoding. It now adds the query as text, and encodes it in the link.
       4. ⚑ edits1 drops repeated candidates with a Set. Array.unique, which it replaces, skipped the element after each one it removed.
       5. ⚑ SuggestJavascriptTest checks the corrections under node. A walk of three queries shows the same on a results page.
   57. ⚑ A page that writes a name with accents ranks as one that writes it without them. www.sjsu.edu, titled San José State University, fell out of the top results for san jose state university.
       1. ⚑ hyphenateEntities looks a name up again with its diacritics folded when it is not found as written. The list of names holds san jose state university without the accent.
       2. ⚑ addExactMatchScore compares the typed word and the result's text with diacritics folded, so jose counts as an exact match for José.
       3. ⚑ On a test index of the four pages from the reported results, the page went from fourth, relevance 14, to first, relevance 21.17.
   58. ⚑ A search's first look at the url signal shares starts from totals saved under the temporary folder. signalShares read every partition's signals file in each new process. It now saves its totals under C\TEMP_DIR and later processes start from them. Saved totals are dropped when the last counted signals file was written again.
   59. ⚑ A full first block scores a word's field key from the word's other key. Once the first block of a union is full, a field key part read its postings in every partition only to add relevance. It now moves past the block, and the other key's walk adds its relevance from the same position list. On the local 300-partition index, postings reads fell from 782 to 576 over ten queries with identical results.
   60. ⚑ A stop reaches a daemon process that misses its lock being gone. An Indexer in a long merge missed a four second stop. It then found the restarted queue server's lock and ran on with the old crawl. Both Indexers wrote the crawl status, so fetchers switched between the two crawls. CrawlDaemon::stop now records when it was asked, and processHandler stops a process that began before then.
   61. ⚑ Sites get the locale resources changed since version 15, and the unit tests run without a deprecation notice. Ten patches changed Tokenizer files without raising RESOURCES_WIKI_VERSION, so a site's own locale copy kept the old ones. The version is now 16, and cut_patch.sh refuses a resources change without a version change. EnTokenizerTest's case checking Tokenizer::$stems tested only the cache against itself, so it is gone. CrawlDaemon's str_getcsv now names its escape character. check_deprecated.sh now finds a csv call that does not.
   62. ⚑ The locale copy that failed EnTokenizerTest was made by code from 28 to 30 September. From commit ad520bf until commit 56dd699, Config.php copied src/locale into a work directory with no app/locale whenever Yioop loaded. A copy made then has a Tokenizer without $stems, at version 15. Gold, still at 15, never refreshed it, and the test stopped at line 73. Rebuilt that way in a scratch tree, the copy failed the gold test the same way. One page served at version 16 brought the copy up to date, and the test then passed. Today's code made no copy over a full unit run, createdb and a page walk.
   63. ⚑ QueryTool says where the site home page named by a query word stands in its results. Searches for yahoo and pepsi do not list www.yahoo.com or www.pepsi.com near the top, though both are in the index. After its statistics QueryTool runs the query again for every result it scores. For each word it prints the place, rank, relevance and score of https://www.word.com/ and https://word.com/. This gives the numbers a ranking change has to be measured against on the index that shows the fault.
   64. ⚑ A site's app/locale holds only its own changes, and programs the web server starts no longer hold its connections. Strings and resources are read from src/locale, with app/locale's changes in their place. A saved configure.ini keeps only strings that differ from the shipped ones. Database version 132 deletes resources...old folders and resources folders holding only copies. Nothing writes to src/locale; TokenTool and Manage Locales write to app/locale. A Launcher, started before the server takes connections, starts the programs the server is asked to start. A fetcher it starts held 8 files and connections, against 967.
   65. ⚑ php GroupWikiTool.php clean deletes unused folders from app/resources. It looks only at folders named by a hash, and their thumbnail folders. A hash is in use if it belongs to a page or post of an existing group, a user, or the feed. Folders reached from one in use by a redirect.txt or a link are kept too. Unused folders, empty hashed folders and empty prefix folders are deleted; a link is deleted without what it points to. If no folder is in use, AUTH_KEY has likely changed, and nothing is deleted.
   66. ⚑ clean deletes prefix folders that look empty. A prefix folder holding only .DS_Store, as Finder leaves, was kept, since it was not strictly empty. .DS_Store, Thumbs.db, desktop.ini and empty dot folders such as .archive no longer count. A page's empty sub-folder, made by a person, still keeps its page folder and prefix folder. A thumbnail folder holding only empty folders goes once its resource folder is gone.
   67. ⚑ A git page in an Invite Only group can be cloned with a user name and Git application code. A clone of a group the public may not read got 404 Not Found. git sends the url's name and code only after a 401 asking for them. GitComponent::checkReader now answers 401 with WWW-Authenticate when no code is accepted. A member of the group, or root, is then served; anyone else gets 404.
   68. ⚑ Every docblock in GitComponent.php names what it documents and explains WebDAV. The class and 26 methods opened with a bare verb, such as Sends or Reports. Each was rewritten by hand to open with its name and say who calls it. WebDAV is explained as the protocol for writing files on a web server wherever it is used.
   69. ⚑ A git page takes pushes over the smart protocol, so a pushed user name and code can come from a prompt. git-http-push, the older dumb push, never asks for a password. It failed with "Cannot access URL ..., return code 22" unless the url held the code. GitReceivePack answers service=git-receive-pack and stores each pushed object as a loose object. It moves a ref only when its old value matches, and reports each command. GitComponent::serveReceivePack asks for an editor's code with a 401 first. settle_gold.sh now moves work's confirmed-head to the work commit matching gold, not to work's HEAD. On HEAD, a settle made mid-round left that round's commits out of the next cut.
   70. ⚑ Each Git repository page has its own Git application code. A code was kept per person, so a fresh code on one page broke every other repository's clone urls. GIT_APP_CODE is now keyed by USER_ID, GROUP_ID and PAGE_ID, the page's id in the default locale. Database version 133 rebuilds the table and keeps each old code with group and page 0. GitModel::getAppCode gives that old code for any page with no code of its own.
   71. ⚑ A git push of tens of megabytes goes through over HTTP/2 and HTTP/1.1. Over HTTP/2 the server copied a request body whole on every frame, so a 40 MB push never finished. Over HTTP/1.1 a chunked body, which git sends past one megabyte, was not read at all. The HTTP/2 body now grows in place, and WebSite::decodeChunkedBody joins chunked bodies. index.php raises the request size limit to the Git push size limit, or lifts it when that is Unlimited. GitReceivePack writes each object as it reads it, keeping only names in memory.
   72. ⚑ A clone or pull copies stored pack entries instead of rebuilding every object. A clone decompressed, rebuilt and recompressed every object, so a large history took minutes before git got a reply. GitPackReader::storedEntry gives an entry still compressed; packChunks copies it when the client can read it. A delta stored by position is sent by its base's name. On a 6,967-commit test repository a clone's reply took 7 s, against 54 s before.
   73. ⚑ An old site upgrades without warnings, and imports a large export of wiki pages. Upgrade 65 made the Search group through GroupModel, which writes to SOCIAL_GROUPS; before upgrade 70 that table is GROUPS. It now inserts into GROUPS, and addGroup gives -1 when a group cannot be stored. updateTranslationForStringId skips a locale the site does not have. WikiPagesFile::eachPage reads a pages file a page at a time, and upgradePublicHelpWiki writes each page as it comes. The shipped pages now import within 24 MB of memory.
   74. ⚑ A clone answered by a web server worker no longer runs the server out of memory. A worker sent its whole reply to the front process as one message. A 334 MB pack then overran a 128 MB limit, and the front stopped. git then saw the connection end with no reply. A worker now hands back the name of the reply's file, and the front sends the file from disk. The HTTP/2 path also deletes such a file once it is sent, as HTTP/1.1 already did.
   75. ⚑ A clone's reply starts reaching git while its pack is still being built. serveUploadPack names its reply file to the server as a growing file once the first piece is written. The server sends what is written, in chunks under HTTP/1.1 and in DATA frames under HTTP/2 and HTTP/3. A deferred file reply over HTTP/3 had been handed to the HTTP/2 sender and sent nothing; it now has its own path. A worker names the file to the front at once, and its last reply ends the send. An HTTP/1.1 connection waiting on the file leaves the write set, so the loop does not spin. On the 6,967-commit test repository git had headers 1.2 to 1.6 s after asking, with and without workers.
   76. ⚑ git fetch --depth deepens a shallow clone, and git fetch --deepen works. Held history was marked past the client's shallow commits, so a deeper fetch found nothing to send. That marking now stops at them, and shallowCutoff works out the new end of history apart from what the client holds. The advertisement offers deepen-relative, which --deepen counts from the client's shallow commits. A depth round that offers held commits now gets its acknowledgments, which --unshallow waited for. Every docblock in GitUploadPack, GitUploadPackTest and WebSiteTest now opens with the name of what it documents. The one-line isObjectName and two one-line test helpers are written where they were used, and an unused test helper is gone.
   77. ⚑ The process file asks for whole-request patches and enforces the nine-minute round. A patch carries every part of the request, including items the work finds the request needs. settle_gold.sh records when a round began, and cut_patch.sh refuses an early cut without SOLVED=1. A reproduction takes the reporting site's worker count and memory limit, and no command reaches an outside host.
   78. ⚑ Resource folders are named from group, page and user names, so they are the same on every instance. A page's files are in <G>/wiki/<locale>/<P>/page and its thumbnails in the thumbs folder beside it. A post's are under <G>/posts/<YYYY-MM>, named from its author and time; feed pictures are in the Public group's feeds folder. Each fixed name is seven readable characters, padded with _, then the raw crawlHash in hexadecimal. Icons are in users/<U>, site files in site, and a host's files in domains/<host>. Personal groups are named <user name>$Personal, and createGroup refuses any name holding a $. A request with s reads only user_icon.webp, and one with neither g nor s only the site or a host folder. Upgrade 134 and ConfigureTool migrate-pre-v10-resources move the pre-version 10 folders and rename personal groups. Running either again does nothing. The migration is the class V10ResourceFolderMigration, and no day-to-day class knows the pre-version 10 layout. Message links are relative, and the clean command follows the version 10 layout.
   79. ⚑ The resource migration deletes prefix folders holding only Finder or Explorer files. A prefix folder holding only .DS_Store, Thumbs.db or desktop.ini was kept, as a UuB folder was on a Mac test site. Such folders now go, and a prefix folder holding anything else is named in the report with what it holds.
   80. ⚑ ConfigureTool's menu has a step that migrates pre-version 10 resource folders. Migrate Pre-Version 10 Resource Folders runs what the migrate-pre-v10-resources command runs. Its report shows as the menu's message.
   81. ⚑ An empty size setting no longer stops the server's session settings. metricToInt read the last letter of an empty GIT_MAX_PUSH_SIZE and printed a warning. That output came before index.php set session.gc_maxlifetime, so PHP refused the setting with a second warning. An empty size now reads as 0, and an empty name server field gets its slash the same way.
   82. ⚑ A slow request no longer leaves the requests queued behind it unanswered. On yioop.com a worker was ended after 6 seconds on a search, with 11 requests queued to it. Those requests then waited on the worker started under the same number, which never got them, for 30 to 359 seconds. A request now waits on its worker's link, not its number. The front holds a request until a worker is free, so none queues behind a slow one. The worker's word that it took a request up is kept apart, so an ended request is answered with a 502, not run again.
   83. ⚑ The web server keeps serving while a worker starts. startWorker waited for a new worker to connect, and the whole front waited with it. A favicon on yioop.com took 3.96 seconds during one such start. finishWorkerStarts now takes a worker in once it connects, on each pass of the event loop. A worker whose front never answers its handshake gives up after WORKER_START_SECONDS, not waiting for good.
   84. ⚑ The supervisor starts the mail server as root. The root supervisor handed MailServer to the Launcher, which runs as WWW_USER. The mail server then could not bind its ports below 1024 and stopped, and was started again every 20 seconds. The Launcher now writes the user it runs as, and only a caller running as that user hands it a program.
   85. ⚑ A start that fails says why. On pollett.org, after a move from Intel to Apple silicon Homebrew, secure-start started nothing and printed nothing. The programs a start launches sent their output to /dev/null, so a php that failed to load said so to no one. Their output now goes to the end of index.log. They are run with the php running the start, not whichever php the path finds. A start of the web server waits up to 15 seconds for it to answer, and otherwise prints the end of index.log.
   86. ⚑ The Launcher writes no log of its own. Its lines go to index.log.
   87. ⚑ No file is kept for the web server's port or the Launcher's user. IndexServerPort.txt and LauncherUser.txt are gone. A start connects to the server's address to see that it came up. It refuses at once when another program answers there. A caller that is not root shares the Launcher's user when it may signal the Launcher's process. Root hands a program over only where there is no WWW_USER.
   88. ⚑ The site's logos, favicon and background are found by the names of their roles. Profile.php no longer holds LOGO_SMALL, LOGO_MEDIUM, LOGO_LARGE, FAVICON or BACKGROUND_IMAGE. An upload is saved in app/resources/site as logo-small, logo-medium, logo-large, favicon or background, with its type's extension. ResourceFolders::siteImageUrl gives that file, or else the image Yioop ships. Database version 135 renames images uploaded under their own names and writes Profile.php again without the five fields.
   89. ✓ A ballot witness under LDAP gives a directory login again. The witness form asked for a login only when USE_LDAP was set, and nothing defines it now. It checks AUTH_METHOD instead, and the emailed-link page asks for the login too. The login must map to that witness's own account, which a new case checks.
   90. ✓ A saved page resource is served as saved. An edited HTML resource kept being served in its old form. The save went through VersionManager, which wrote with file_put_contents. WikiModel now gives VersionManager its filePutContents, and WebSite::filePutContents drops the gzip copy of a file it writes.
   91. ⚑ The unit tests write only inside the tests folder. A run under strace found 53 test files writing the site's work directory, its databases, /tmp or /root.
       1. ⚑ The tests run in a work directory of their own. CodeTool's unit command uses tests/test_files/test_work_directory. Its profile names two SQLite databases there, whatever database the site uses, and createdb fills them on the first run. Config.php takes that folder from YIOOP_TEST_WORK_DIRECTORY on the command line.
       2. ⚑ Tests keep their own files in tests/test_files. Files made in /tmp or the work directory now go there. openssl's random seed goes there through RANDFILE, and the test web servers keep sessions in C\TEMP_DIR.
       3. ⚑ check_test_writes.sh names each write outside tests. It runs each test file under strace and prints full paths. CrawlDaemonTest no longer runs php -r, and MessagesBundleTest no longer runs rm -rf.
       4. ⚑ The tests' profile parses when the site's database address holds a semicolon. UnitTest::testWorkProfile replaces each database setting's whole line. Replacing up to the first semicolon broke an address such as mysql:host=localhost;dbname=yioop.
       5. ⚑ The tests' databases are copied again when they are empty or old. They are copied from the two databases Yioop ships in src/data and upgraded to DATABASE_VERSION. That takes about two seconds, where createdb took 43. A run only checked that the file was there, so a run that stopped left empty databases every later run used. ProfileModel::testDatabasesMade checks the version and tables first. The unit command upgrades databases only in the tests' work directory. Listing the tests leaves the site's databases as they are.
       6. ⚑ A request waiting while every worker is replaced waits for the new one. waitForFreeWorker answered 503 when all workers had been asked to stop. It now waits until one is started in their place.
   92. ⚑ The unit tests start no processes and use the site's work directory.
       1. ⚑ The tests' own work directory is gone. The tests use C\WORK_DIRECTORY, C\TEMP_DIR and C\TEST_DIR again. With the second process that set it up gone, results are colored again.
       2. ⚑ No test starts a process. UnitTest::startMethod and the unit command's child mode are gone. CsrBuilderTest, ArcToolTest and AVToolTest now call the code in their own process.
       3. ⚑ Cases that need a second process are removed. They are three UndoLog stop cases, the feed stop case and the queue server message case. Also four CrawlDaemon cases, seven FetchUrl server cases and five WebSite worker cases.
       4. ⚑ Tests write only in the tests folder. Code a test reaches takes the folder it writes in as a property, such as CrawlDaemon::$schedules_folder and ProfileModel::$resource_path. Tests that used the site's databases make their own in tests/test_files.
       5. ⚑ UndoLog syncs with fsync only. It ran sync -f on Linux. The tesseract case of PdfProcessorTest is removed.
       6. ⚑ Folder lookups that can give false are not taken apart. PHP 8.5 warns when list() takes apart false. deleteGroupPage and two other callers now give an array in its place, and check_deprecated.sh names such statements.
       7. ⚑ The unit tests run faster and print as they go. The suite took 38 seconds, with 66 cases over a tenth of a second. It now takes about 12 seconds, and the cases still over that limit are listed for Chris to decide on.
          1. ⚑ CodeTool.php unit prints each test class's results as soon as the class has run.
          2. ⚑ Key and password cases use smaller RSA keys and the cheapest bcrypt cost. Production sizes and costs are unchanged.
          3. ⚑ The storage cases use fewer rows and trees, and each still checks the same behavior. Two waits on a clock take a twentieth of a second.
          4. ⚑ namesOfferedExtend WhatWasTypedTestCase uses its own test database. It failed on a site with five users whose names start with "ro" before "root".
          5. ⚑ AVToolTest no longer checks that a still picture is the default. Chris removed it as checking one line of option handling.
          6. ⚑ indexServerUpWaitsFor AnAnswerTestCase checks that the wait writes no file of ports. It failed on a site still holding IndexServerPort.txt from before.